Image control system

ABSTRACT

An image control system includes a rear camera, a left rear (right rear) camera, a display device, and an image control device for generating a combined image and displaying the combined image on the display device. The image control device identifies a rear target object and a left rear (right rear) target object which are closest to the vehicle in an own lane and a left lane (right lane), respectively, and changes a magnitude of the rear left-side (right-side) azimuth angle being an azimuth angle of a left (right) boundary line defining a horizontal angle of view of a rear effective range based on a combination of presence or absence of the rear target object and a level of closeness to this object and presence or absence of the left rear (right rear) target object and a level of closeness to this object.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to Japanese Patent Application No.2021-177843 filed on Oct. 29, 2021, which is incorporated herein byreference in its entirety including the description, claims, drawings,and abstract.

BACKGROUND 1. Field

The present disclosure relates to an image control system for generatinga panorama-format combined image from pieces of image data acquired by aplurality of image pickup devices, and displaying the combined image ona display device.

2. Description of the Related Art

Hitherto, there has been known a technology of generating apanorama-format combined image from pieces of image data acquired by aplurality of image pickup devices (typically, in-vehicle cameras), anddisplaying the combined image on a display device. For example, an imagedisplay control apparatus as described in Japanese Patent ApplicationLaid-open No. 2016-189576 (hereinafter also referred to as “related-artapparatus”) acquires a first rear image, a second rear image, and athird rear image. The first rear image is obtained by picking up animage of a rear side of a vehicle body from a rear portion of thevehicle body. The second rear image is obtained by picking up an imageof a right rear side of the vehicle body. The third rear image isobtained by picking up an image of a left rear side of the vehicle body.

When the related-art apparatus determines that an object is detectedbased on those rear images, the related-art apparatus converts each ofthe first to third rear images so that, through use of a homographycalculated based on a distance from the rear portion of the vehicle bodyto the object, an angle of view of the first rear image is increased asthis distance is decreased. Then, the related-art apparatus combines theconverted rear images so as to generate a panorama-format combined imageviewed from a rear end portion of the vehicle body as a viewpoint, anddisplays the combined image on the display device. In Japanese PatentApplication Laid-open No. 2016-189576, the object is described as“obstacle.”

In Japanese Patent Application Laid-open No. 2016-189576, it isdescribed that, according to the related-art apparatus, the entireobject can be displayed in the combined image with seams of the combinedimage being smooth. However, in the configuration of the related-artapparatus, when a plurality of objects are detected, there is apossibility that those objects cannot be appropriately displayed in thecombined image. That is, the related-art apparatus is configured toconvert, for one detected object, the first to third rear images so thatthe angle of view of the first rear image is increased as the distancefrom the rear portion of the vehicle body to this object is decreased.Accordingly, it is considered that for this object, the entire objectcan be displayed when the above-mentioned conversion processing isperformed. However, in Japanese Patent Application Laid-open No.2016-189576, there is no discussion about how to convert each rear imagewhen other objects are detected. Thus, there is a possibility that thoseother objects are not appropriately displayed in the combined image, andas a result, there is a possibility that reduction in traveling safetyof the vehicle is caused.

However, when it is attempted to appropriately display all of thedetected objects in the combined image, there is a possibility that theconversion processing becomes complicated and a processing load of therelated-art apparatus is consequently increased.

SUMMARY

The present disclosure has been made in order to cope with theabove-mentioned problems. That is, the present disclosure has an objectto provide an image control system capable of achieving both ofappropriate display of an object having a relatively high possibility ofaffecting traveling of a vehicle in a combined image, and suppression ofan increase of a processing load applied when the combined image isgenerated.

According to at least one embodiment of the present disclosure, there isprovided an image control system (hereinafter also referred to as “thepresent system”) including: a rear image pickup device (11Re) configuredto pick up images of an object and a dividing line which are present ina rear image pickup range (Rre) expanding on a rear side of a vehicle(V); a left rear image pickup device (11L) configured to pick up imagesof an object and a dividing line which are present in a left rear imagepickup range (RI) expanding on a left rear side of the vehicle andpartially overlapping with the rear image pickup range; a right rearimage pickup device (11R) configured to pick up images of an object anda dividing line which are present in a right rear image pickup range(Rr) expanding on a right rear side of the vehicle and partiallyoverlapping with the rear image pickup range; a display device (12)including a display screen (12 a); and an image control device (10)configured to: acquire rear image data obtained by picking up the imagesby the rear image pickup device, left rear image data obtained bypicking up the images by the left rear image pickup device, and rightrear image data obtained by picking up the images by the right rearimage pickup device; generate a combined image having a panorama formatbased on the rear image data, the left rear image data, and the rightrear image data; and display the combined image on the display screen ofthe display device.

When, in the rear image pickup range, the left rear image pickup range,and the right rear image pickup range, image pickup ranges to be usedfor generating the combined image are defined as a rear effective range(Rrea), a left rear effective range (Rla), and a right rear effectiverange (Rra), respectively, when, in a case in which a direction of anoptical axis (Are) of the rear image pickup device is set as a rearreference direction, among boundary lines (30 la, 30 ra) defining ahorizontal angle of view of the rear effective range, an angle from therear reference direction of a rear left-side boundary line (30 la) beingthe boundary line on a left side of the vehicle is defined as a rearleft-side azimuth angle (θl) and an angle from the rear referencedirection of a rear right-side boundary line (30 ra) being the boundaryline on a right side of the vehicle is defined as a rear right-sideazimuth angle (θr), respectively, when, in a case in which a directionof an optical axis (Al) of the left rear image pickup device is set as aleft rear reference direction, among boundary lines (40 ia, 40 oa)defining a horizontal angle of view of the left rear effective range, anangle from the left rear reference direction of a left rear inner-sideboundary line (40 ia) being the boundary line on an inner side in avehicle width direction of the vehicle is defined as a left rearinner-side azimuth angle (θli), and when, in a case in which a directionof an optical axis (Ar) of the right rear image pickup device is set asa right rear reference direction, among boundary lines (50 ia, 50 oa)defining a horizontal angle of view of the right rear effective range,an angle from the right rear reference direction of a right rearinner-side boundary line (50 ia) being the boundary line on an innerside in the vehicle width direction of the vehicle is defined as a rightrear inner-side azimuth angle (θri), the image control device (10) isconfigured to: allow the rear effective range (Rrea) to be changed bychanging magnitudes of the rear left-side azimuth angle (θl) and therear right-side azimuth angle (θr) independently of each other within arange of a horizontal angle of view of the rear image pickup device;allow the left rear effective range (Rla) to be changed by changing amagnitude of the left rear inner-side azimuth angle (θli) within a rangeof a horizontal angle of view of the left rear image pickup device;allow the right rear effective range (Rra) to be changed by changing amagnitude of the right rear inner-side azimuth angle (θri) within arange of a horizontal angle of view of the right rear image pickupdevice; and calculate, based on the dividing line (d1, d2, d3, d4)detected from each of the rear image data, the left rear image data, andthe right rear image data, a position and a shape of each of an own lane(L) on which the vehicle is positioned, a left lane (LI) adjacent to theown lane on the left side thereof, and a right lane (Lr) adjacent to theown lane on the right side thereof.

When the image control device is to generate the combined image, theimage control device is configured to: identify a rear target object, aleft rear target object, and a right rear target object which areobjects closest to the vehicle in the own lane, the left lane, and theright lane, respectively, among the objects positioned on the rear sidewith respect to a rear end portion of the vehicle, which are detectedfrom the rear image data, the left rear image data, and the right rearimage data; change the magnitude of the rear left-side azimuth angle(θl) based on a combination of presence or absence of the rear targetobject and a level of closeness to the rear target object and presenceor absence of the left rear target object and a level of closeness tothe left rear target object; change the magnitude of the rear right-sideazimuth angle (θr) based on a combination of the presence or absence ofthe rear target object and the level of closeness to the rear targetobject and presence or absence of the right rear target object and alevel of closeness to the right rear target object; change, when themagnitude of the rear left-side azimuth angle (θl) or the rearright-side azimuth angle (θr) is changed to a relatively small value,the magnitude of the left rear inner-side azimuth angle (θli) or theright rear inner-side azimuth angle (θri) to a relatively large value,respectively, so as to generate the combined image; and change, when themagnitude of the rear left-side azimuth angle or the rear right-sideazimuth angle is changed to a relatively large value, the magnitude ofthe left rear inner-side azimuth angle or the right rear inner-sideazimuth angle to a relatively small value, respectively, so as togenerate the combined image.

In the present system, when the combined image is to be generated, themagnitude of the rear left-side azimuth angle (angle from the rearreference direction of the boundary line on the left side of the vehicle(rear left-side boundary line) among the boundary lines defining thehorizontal angle of view of the rear effective range) is changed basedon the combination of the presence or absence of the rear target object(object closest to the vehicle in the own lane) and the level ofcloseness to the rear target object (for example, any of “close,”“intermediate,” or “far”) and the presence or absence of the left reartarget object (object closest to the vehicle in the left lane) and thelevel of closeness to the left rear target object. In addition, when themagnitude of the rear left-side azimuth angle is changed, the magnitudeof the left rear inner-side azimuth angle (angle from the left rearreference direction of the boundary line on the inner side in thevehicle width direction (left rear inner-side boundary line) among theboundary lines defining the horizontal angle of view of the left reareffective range) is changed in accordance with this change. When each ofthe magnitude of the rear left-side azimuth angle and the magnitude ofthe left rear inner-side azimuth angle is appropriately changed based onthe above-mentioned combination, the rear target object and the leftrear target object (when only one of the target objects is detected, thedetected object) can be appropriately displayed in the combined image.

Similarly, in the present system, when the combined image is to begenerated, the magnitude of the rear right-side azimuth angle (anglefrom the rear reference direction of the boundary line on the right sideof the vehicle (rear right-side boundary line) among the boundary linesdefining the horizontal angle of view of the rear effective range) ischanged based on the combination of the presence or absence of the reartarget object and the level of closeness to the rear target object andthe presence or absence of the right rear target object (object closestto the vehicle in the right lane) and the level of closeness to theright rear target object. In addition, when the magnitude of the rearright-side azimuth angle is changed, the magnitude of the right rearinner-side azimuth angle (angle from the right rear reference directionof the boundary line on the inner side in the vehicle width direction(right rear inner-side boundary line) among the boundary lines definingthe horizontal angle of view of the right rear effective range) ischanged in accordance with this change. When each of the magnitude ofthe rear right-side azimuth angle and the magnitude of the right rearinner-side azimuth angle is appropriately changed based on theabove-mentioned combination, the rear target object and the right reartarget object (when only one of the target objects is detected, thedetected object) can be appropriately displayed in the combined image.

In this case, among the objects positioned on the rear side of thevehicle, object having a relatively high possibility of affecting thetraveling of the vehicle is an object closest to the vehicle in each ofthe own lane, the left lane, and the right lane (that is, the reartarget object, the left rear target object, and the right rear targetobject). Accordingly, with the configuration of the present system, theobject having a relatively high possibility of affecting the travelingof the vehicle can be appropriately displayed in the combined image.

In addition, the changes of the rear left-side azimuth angle and theleft rear inner-side azimuth angle and the changes of the rearright-side azimuth angle and the right rear inner-side azimuth angle areperformed based on only information on the rear target object, the leftrear target object, and the right rear target object (that is, thepresence or absence of those target objects and the level of closenessto those target objects), and information on other objects is notconsidered. Thus, an increase of a processing load applied when thecombined image is generated can be suppressed.

According to one aspect of the present disclosure, the image controldevice (10) is configured to: allow the rear left-side azimuth angle(θl) to be changed between a first rear left-side angle (θn) and asecond rear left-side angle (θw) having a magnitude larger than amagnitude of the first rear left-side angle; allow the rear right-sideazimuth angle (θr) to be changed between a first rear right-side angle(−θn) and a second rear right-side angle (−θw) having a magnitude largerthan a magnitude of the first rear right-side angle; allow the left rearinner-side azimuth angle (θli) to be changed between a first left rearinner-side angle (−θ1) and a second left rear inner-side angle (−θ2)having a magnitude larger than a magnitude of the first left rearinner-side angle; and allow the right rear inner-side azimuth angle(θri) to be changed between a first right rear inner-side angle (θ1) anda second right rear inner-side angle (θ2) having a magnitude larger thana magnitude of the first right rear inner-side angle.

With this configuration, with a relatively simple configuration, therear target object, the left rear target object, and the right reartarget object can be appropriately displayed in the combined image.

According to one aspect of the present disclosure, the image controldevice (10) is configured to: set an imaginary projection surface (Sp)orthogonal to a tread surface of the vehicle (V) at a position separatedaway from the rear end portion of the vehicle rearward by apredetermined projection distance (D3) determined as a fixed value; andgenerate, as the combined image, an image obtained by projecting, ontothe imaginary projection surface, the object and the dividing linepresent in each of the rear effective range (Rrea), the left reareffective range (Rla), and the right rear effective range (Rra) throughuse of a viewpoint (Pv) as a reference, the viewpoint (Pv) beingimaginarily set on the vehicle or in vicinity thereof, and, when theimage control device is to generate the combined image, the imagecontrol device is configured to change each of the rear left-sideazimuth angle (θl), the left rear inner-side azimuth angle (θli), therear right-side azimuth angle (θr), and the right rear inner-sideazimuth angle (θri) so that a left-side intersection (Pl) and aright-side intersection (Pr) are positioned on the imaginary projectionsurface in plan view of the vehicle, the left-side intersection (Pl)being an intersection between the rear left-side boundary line (30 la)and the left rear inner-side boundary line (40 ia), the right-sideintersection (Pr) being an intersection between the rear right-sideboundary line (30 ra) and the right rear inner-side boundary line (50ia).

According to the one aspect of the present disclosure, the left-sideintersection (intersection between the rear left-side boundary line andthe left rear inner-side boundary line) is positioned on the projectionsurface in plan view of the vehicle. Accordingly, an overlapping part(overlapping region) between the rear effective range and the left reareffective range can be suppressed to be as small as possible, and arange in which the object and/or the dividing line is displayed doublyin the combined image can be reduced to be as small as possible.Similarly, in the present system, the right-side intersection(intersection between the rear right-side boundary line and the rightrear inner-side boundary line) is positioned on the projection surfacein plan view of the vehicle. Accordingly, an overlapping part(overlapping region) between the rear effective range and the right reareffective range can be suppressed to be as small as possible, and arange in which the object and/or the dividing line is displayed doublyin the combined image can be reduced to be as small as possible. In thefollowing, a phenomenon in which the object and/or the dividing line isdisplayed doubly in the combined image is also referred to as “doublingphenomenon.”

The left-side intersection is positioned at a seam between the rearimage data and the left rear image data (strictly speaking, between theimage data corresponding to the rear effective range in the rear imagedata and the image data corresponding to the left rear effective rangein the left rear image data) in the combined image. Similarly, theright-side intersection is positioned at a seam between the rear imagedata and the right rear image data (strictly speaking, between the imagedata corresponding to the rear effective range in the rear image dataand the image data corresponding to the right rear effective range inthe right rear image data) in the combined image.

In this case, when a region between a first straight line (21) passingthrough the rear end portion of the vehicle (V) and extending in thevehicle width direction of the vehicle and a second straight line (22)separated away from the first straight line rearward by a predeterminedfirst distance (D1) and in parallel to the first straight line isdefined as a first zone (Z1), a region between the second straight lineand a third straight line (23) separated away from the first straightline rearward by a predetermined second distance (D2) and in parallel tothe first straight line is defined as a second zone (Z2), and a regionon the rear side with respect to the third straight line (23) is definedas a third zone (Z3), the predetermined projection distance (D3) islonger than the predetermined first distance and is shorter than thepredetermined second distance, and when an intersection between theimaginary projection surface (Sp) and a left imaginary line extendingrearward from a left rear corner portion (C1) of the vehicle so as to beparallel to a longitudinal axis (A) of the vehicle is defined as a firstintersection (P11), and an intersection between the imaginary projectionsurface (Sp) and a right imaginary line extending rearward from a rightrear corner portion (Cr) of the vehicle so as to be parallel to thelongitudinal axis (A) of the vehicle is defined as a second intersection(P12), in a case in which the magnitude of the rear left-side azimuthangle (θl) or the rear right-side azimuth angle (θr) is changed to therelatively small value, in plan view of the vehicle, the left-sideintersection (Pl) or the right-side intersection (Pr) is positioned onthe first intersection or in vicinity thereof, or on the secondintersection or in vicinity thereof, respectively, and in a case inwhich the magnitude of the rear left-side azimuth angle or the rearright-side azimuth angle is changed to the relatively large value, theleft-side intersection or the right-side intersection is positioned at aposition separated away from the first intersection or the secondintersection outward in the vehicle width direction of the vehicle,respectively, by a predetermined distance equal to or larger than ageneral lane width.

With this configuration, no matter what combination the above-mentionedcombination of the rear target object and the left rear target object(or the right rear target object) may be, the rear target object, theleft rear target object, and the right rear target object can beappropriately displayed in the combined image.

In this case, the predetermined projection distance (D3) is smaller than½ of a sum of the predetermined first distance (D1) and thepredetermined second distance (D2).

With this configuration, no matter what combination the above-mentionedcombination of the rear target object and the left rear target object(or the right rear target object) may be, the rear target object, theleft rear target object, and the right rear target object can be moreappropriately displayed in the combined image.

According to one aspect of the present disclosure, the image controldevice (10) is configured to: change the magnitude of the rear left-sideazimuth angle (θl) to the relatively large value when the rear targetobject is positioned in the first zone (Z1) and the left rear targetobject is positioned on the rear side with respect to the first zone;and change the magnitude of the rear right-side azimuth angle (θr) tothe relatively large value when the rear target object is positioned inthe first zone and the right rear target object is positioned on therear side with respect to the first zone.

In a related-art image control system, there is a possibility that theleft rear target object (or the right rear target object) is hiddenbehind the rear target object depending on the positional relationshipbetween the rear target object and the left rear target object (or theright rear target object) and that the left rear target object (or theright rear target object) is not displayed in the combined image (in thefollowing, this phenomenon is also referred to as “hiding phenomenon”).Meanwhile, when the magnitude of the rear left-side azimuth angle (orthe rear right-side azimuth angle) is changed to the relatively smallvalue, there is a possibility that a part of the rear target object(typically, a front corner portion thereof) is cut off from the combinedimage (in the following, this phenomenon is also referred to as “cut-offphenomenon”). That is, there is a possibility that a turn signal lamp ofthe rear target object is not displayed in the combined image, and, as aresult, a driver of the vehicle cannot determine from the combined imagewhether or not the rear target object has an intention to change thelane. When the vehicle changes the lane without noticing the intentionto change the lane in spite of the rear target object having theintention, there is a possibility that the rear target object comesexcessively close to or comes into contact with the vehicle, and hencethe traveling safety of the vehicle is reduced.

In view of the above, according to the one aspect of the presentdisclosure, when the rear target object is positioned in the first zoneand the left rear target object (or the right rear target object) ispositioned on the rear side with respect to the first zone, themagnitude of the rear left-side azimuth angle (or the rear right-sideazimuth angle) is changed to the relatively large value. That is,priority is given more to solving the cut-off phenomenon of the reartarget object than to solving the hiding phenomenon of the left reartarget object (or the right rear target object). With thisconfiguration, at least the rear target object can be appropriatelydisplayed in the combined image, and hence reduction of the travelingsafety of the vehicle can be suppressed.

According to one aspect of the present disclosure, the image controldevice (10) is configured to: change the magnitude of the rear left-sideazimuth angle (θl) to the relatively large value when the rear targetobject is positioned in the first zone (Z1) and the left rear targetobject is absent; and change the magnitude of the rear right-sideazimuth angle (θr) to the relatively large value when the rear targetobject is positioned in the first zone and the right rear target objectis absent.

When the magnitude of the rear left-side azimuth angle (or the rearright-side azimuth angle) is changed to the relatively small value, thecut-off phenomenon occurs in the rear target object, and, in some cases,the vehicle may change the lane while the driver of the vehicle does notnotice that the rear target object has an intention to change the lane.In this case, there is a possibility that the rear target object comesexcessively close to or comes into contact with the vehicle, and hencethe traveling safety of the vehicle is reduced. In view of the above,according to the one aspect of the present disclosure, when the reartarget object is positioned in the first zone and the left rear targetobject (or the right rear target object) is absent, the magnitude of therear left-side azimuth angle (or the rear right-side azimuth angle) ischanged to the relatively large value. With this configuration, the reartarget object can be appropriately displayed in the combined image, andhence reduction of the traveling safety of the vehicle can besuppressed.

According to one aspect of the present disclosure, the image controldevice (10) is configured to: change the magnitude of the rear left-sideazimuth angle (θl) to the relatively small value when both of the reartarget object and the left rear target object are positioned in thefirst zone (Z1); and change the magnitude of the rear right-side azimuthangle (θr) to the relatively small value when both of the rear targetobject and the right rear target object are positioned in the firstzone.

When the magnitude of the rear left-side azimuth angle (or the rearright-side azimuth angle) is changed to the relatively small value, theleft rear target object (or the right rear target object) may beappropriately displayed in the combined image, but the cut-offphenomenon may occur in the rear target object, and in some cases, thevehicle may change the lane while the driver of the vehicle does notnotice that the rear target object has an intention to change the lane.In this case, there is a possibility that the rear target object comesexcessively close to or comes into contact with the vehicle, and hencethe traveling safety of the vehicle is reduced. Meanwhile, when themagnitude of the rear left-side azimuth angle (or the rear right-sideazimuth angle) is changed to the relatively large value, the cut-offphenomenon of the rear target object is solved, but there is apossibility that the left rear target object (or the right rear targetobject) is not displayed in the combined image (disappears in thecombined image) because the left rear target object (or the right reartarget object) is positioned in a blind spot region between the reareffective range and the left rear effective range (or the right reareffective range) (in the following, this phenomenon is also referred toas “disappearing phenomenon”). When the vehicle changes the lane whilethe driver does not notice the presence of the left rear target object(or the right rear target object) due to the disappearing phenomenon,there is a possibility that the left rear target object (or the rightrear target object) traveling straight comes into contact with thevehicle, and hence the traveling safety of the vehicle is reduced.

In this case, even when the cut-off phenomenon occurs in the rear targetobject, a part of the rear target object is displayed in the combinedimage, and hence the driver can recognize the presence of the reartarget object itself. In contrast, when the disappearing phenomenonoccurs in the left rear target object (or the right rear target object),this target object is not displayed at all in the combined image, andhence the driver cannot recognize the presence of this target object. Inview of the above, according to the one aspect of the presentdisclosure, when both of the rear target object and the left rear targetobject (or the right rear target object) are positioned in the firstzone, the magnitude of the rear left-side azimuth angle (or the rearright-side azimuth angle) is changed to the relatively small value. Thatis, priority is given more to solving the disappearing phenomenon of theleft rear target object (or the right rear target object) than tosolving the cut-off phenomenon of the rear target object. With thisconfiguration, the left rear target object (or the right rear targetobject) can be appropriately displayed in the combined image, and hencereduction of the traveling safety of the vehicle can be suppressed.

According to one aspect of the present disclosure, the image controldevice (10) is configured to: change the magnitude of the rear left-sideazimuth angle (θl) to the relatively large value when both of the reartarget object and the left rear target object are positioned in thethird zone (Z3); change the magnitude of the rear right-side azimuthangle (θr) to the relatively large value when both of the rear targetobject and the right rear target object are positioned in the thirdzone; change the magnitude of the rear left-side azimuth angle (θl) tothe relatively large value when the rear target object is absent and theleft rear target object is positioned in the third zone; and change themagnitude of the rear right-side azimuth angle (θr) to the relativelylarge value when the rear target object is absent and the right reartarget object is positioned in the third zone.

With this configuration, no doubling phenomenon occurs in the left reartarget object (or the right rear target object), and hence theappearance of the combined image is improved.

According to one aspect of the present disclosure, when the imagecontrol device (10) is to generate the combined image, the image controldevice (10) is configured to: change the rear left-side azimuth angle(θl) and the left rear inner-side azimuth angle (θli) continuously whenchanging the rear left-side azimuth angle (θl) and the left rearinner-side azimuth angle (θli); and change the rear right-side azimuthangle (θr) and the right rear inner-side azimuth angle (θri)continuously when changing the rear right-side azimuth angle (θr) andthe right rear inner-side azimuth angle (θri).

With this configuration, movement of a seam of the combined imagebecomes smoother, and hence switching of the combined image can beperformed smoothly.

In the description above, in order to facilitate understanding of thedisclosure, reference symbols used in at least one embodiment of thepresent disclosure are enclosed in parentheses, and are assigned to eachof constituent features of the disclosure corresponding to the at leastone embodiment. However, each of the constituent features of thedisclosure is not limited to the at least one embodiment prescribed bythe reference symbols.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic configuration diagram of an image control systemaccording to at least one embodiment of the present disclosure.

FIG. 2 is a plan view for illustrating an installation position and animage pickup range of each camera included in the image control system.

FIG. 3 is a flow chart for illustrating a routine to be executed by aCPU of an image control ECU included in the image control system.

FIG. 4 is a plan view for illustrating relative positions of othervehicles with respect to an own vehicle.

FIG. 5A is a plan view for illustrating effective ranges Rrea, Rla, andRra and a projection surface in a case in which azimuth angles θl and θrare changed to angles θn and −θn, respectively.

FIG. 5B is a plan view for illustrating the effective ranges Rrea, Rla,and Rra and the projection surface in a case in which the azimuth anglesθl and θr are changed to angles θw and −θw, respectively.

FIG. 6A is a matrix M1 to be used for determination of the azimuth angleθl.

FIG. 6B is a matrix M2 to be used for determination of the azimuth angleθr.

FIG. 7A is a plan view for illustrating a state in which, in a case ofθl=θn, a target object is absent on an own lane, and a target object ofa left lane is positioned in each zone.

FIG. 7B is a plan view for illustrating a state in which, in a case ofθl=θw, the target object is absent on the own lane, and the targetobject of the left lane is positioned in each zone.

FIG. 8A is a plan view for illustrating a state in which, in the case ofθl=θn, the target object of the own lane is positioned in a zone Z3, andthe target object of the left lane is positioned in each zone.

FIG. 8B is a plan view for illustrating a state in which, in the case ofθl=θw, the target object of the own lane is positioned in the zone Z3,and the target object of the left lane is positioned in each zone.

FIG. 9A is a plan view for illustrating a state in which, in the case ofθl=θn, the target object of the own lane is positioned in a zone Z2, andthe target object of the left lane is positioned in each zone.

FIG. 9B is a plan view for illustrating a state in which, in the case ofθl=θw, the target object of the own lane is positioned in the zone Z2,and the target object of the left lane is positioned in each zone.

FIG. 10A is a plan view for illustrating a state in which, in the caseof θl=θn, the target object of the own lane is positioned in a zone Z1,and the target object of the left lane is positioned in each zone.

FIG. 10B is a plan view for illustrating a state in which, in the caseof θl=θw, the target object of the own lane is positioned in the zoneZ1, and the target object of the left lane is positioned in each zone.

FIG. 11 is a plan view for illustrating an image pickup range of animage control system serving as a first comparative example, and is aview to be used for describing a cut-off phenomenon of an object in acombined image.

FIG. 12 is a plan view for illustrating the image pickup range of theimage control system serving as the first comparative example, and is aview to be used for describing a doubling phenomenon of the object inthe combined image.

FIG. 13 is a plan view for illustrating an image pickup range of animage control system serving as a second comparative example, and is aview to be used for describing a disappearing phenomenon and a hidingphenomenon of the object in the combined image.

FIG. 14A is a matrix M3 serving as a modification example to be used fordetermination of the azimuth angle θl.

FIG. 14B is a matrix M4 serving as another modification example to beused for determination of the azimuth angle θr.

DESCRIPTION OF THE EMBODIMENTS

(Configuration)

FIG. 1 is a schematic configuration diagram of an image control system 1according to at least one embodiment of the present disclosure. Asillustrated in FIG. 1 , the image control system 1 includes an imagecontrol ECU 10 serving as an image control device, a camera 11, and adisplay device 12. The camera 11 and the display device 12 are connectedto the image control ECU 10. The image control ECU 10 includes amicrocomputer as a main unit. The ECU is an abbreviation for electroniccontrol unit. The microcomputer includes a CPU, a ROM, a RAM, aninterface (I/F), and the like, and the CPU implements various functionsby executing instructions (programs or routines) stored in the ROM. Inthe following, a vehicle having the image control system 1 mountedthereon is referred to as “own vehicle.” Further, the image control ECU10 is also simply referred to as “ECU 10.”

The camera 11 includes a rear camera 11Re serving as a rear image pickupdevice, a left side camera 11L serving as a left rear image pickupdevice, and a right side camera 11R serving as a right rear image pickupdevice. As illustrated in FIG. 2 , the rear camera 11Re is installed ata rear-end center portion of an own vehicle V so that an optical axisAre thereof is substantially parallel to a tread surface of the ownvehicle V. That is, the optical axis Are is parallel to a longitudinalaxis A of the own vehicle V. However, an extending direction of theoptical axis Are is not limited thereto. For example, the rear camera11Re may be installed so that the optical axis Are is inclined slightlydownward.

The left side camera 11L and the right side camera 11R are installed atlower portions of left and right side mirrors of the own vehicle V,respectively, so that optical axes Al and Ar thereof are substantiallyparallel to the tread surface of the own vehicle V. However, extendingdirections of the optical axes Al and Ar are not limited thereto. Forexample, the left side camera 11L and the right side camera 11R may beinstalled so that the optical axes Al and Ar are each inclined slightlydownward.

In the following, the rear camera 11Re, the left side camera 11L, andthe right side camera 11R are also referred to as “camera 11Re, camera11L, and camera 11R,” respectively.

Installation positions of the cameras 11L and 11R are not limited to theabove-mentioned positions. For example, the cameras 11L and 11R may bebuilt into the left and right side mirrors, respectively.

The camera 11Re picks up an image of a landscape on a rear side(strictly speaking, on a directly rear side and diagonally rearward leftand right sides) of the own vehicle V. A range Rre shows a plan view ofan image pickup range (rear image pickup range expanding on the rearside of the own vehicle V) of the camera 11Re. The image pickup rangeRre has a fan shape from the position of the camera 11Re serving as acenter, and is line-symmetric with respect to the longitudinal axis A. Ahorizontal angle of view and a vertical angle of view of the camera 11Reare set in advance to such values that an object (typically, anothervehicle or a two-wheeled motor vehicle) and a dividing line positionedon the rear side of the own vehicle V may be included in the imagepickup range Rre. In the at least one embodiment, the horizontal angleof view of the camera 11Re is set to 44 degrees, but the horizontalangle of view is not particularly limited. A boundary line 30 l and aboundary line 30 r are a boundary line on the left side of the ownvehicle V and a boundary line on the right side of the own vehicle V,respectively, among boundary lines defining the horizontal angle of viewof the camera 11Re. In the following, the boundary lines 30 l and 30 rare also simply referred to as “boundary lines 30 l and 30 r of theimage pickup range Rre.” The camera 11Re picks up an image of a subject(typically, an object and a dividing line) present in the image pickuprange Rre, and outputs the obtained image data to the ECU 10 as rearimage data.

The camera 11L picks up an image of a landscape on a left rear side(strictly speaking, on left rear and lateral side) of the own vehicle V.A range Rl shows a plan view of an image pickup range (left rear sideimage pickup range expanding on the left rear side of the own vehicle V)of the camera 11L. The image pickup range Rl has a fan shape from theposition of the camera 11L serving as a center. A horizontal angle ofview and a vertical angle of view of the camera 11L are set in advanceto such values that an object and a dividing line positioned on the leftrear side of the own vehicle V may be included in the image pickup rangeRl. In the at least one embodiment, the horizontal angle of view of thecamera 11L is set to 45 degrees, but the horizontal angle of view is notparticularly limited. A boundary line 40 i and a boundary line 40 o area boundary line on an inner side in a vehicle width direction of the ownvehicle V and a boundary line on an outer side in the vehicle widthdirection of the own vehicle V, respectively, among boundary linesdefining the horizontal angle of view of the camera 11L. In thefollowing, the boundary lines 40 i and 40 o are also simply referred toas “boundary lines 40 i and 40 o of the image pickup range Rl.” Theboundary line 40 i is parallel to the longitudinal axis A, but anextending direction of the boundary line 40 i is not limited thereto.The image pickup range Rl partially overlaps with the image pickup rangeRre (see the shaded part). The camera 11L picks up an image of a subjectpositioned in the image pickup range Rl, and outputs the obtained imagedata to the ECU 10 as left rear image data.

The camera 11R picks up an image of a landscape on a right rear side(strictly speaking, on a right rear and lateral side) of the own vehicleV. A range Rr shows a plan view of an image pickup range (right rearside image pickup range expanding on the right rear side of the ownvehicle V) of the camera 11R. The image pickup range Rr has a fan shapefrom the position of the camera 11R serving as a center. A horizontalangle of view and a vertical angle of view of the camera 11R are set inadvance to such values that an object and a dividing line positioned onthe right rear side of the own vehicle V may be included in the imagepickup range Rr. In the at least one embodiment, the horizontal angle ofview of the camera 11R is set to 45 degrees, but the horizontal angle ofview is not particularly limited. A boundary line 50 i and a boundaryline 50 o are a boundary line on the inner side in the vehicle widthdirection of the own vehicle V and a boundary line on the outer side inthe vehicle width direction of the own vehicle V, respectively, amongboundary lines defining the horizontal angle of view of the camera 11R.In the following, the boundary lines 50 i and 50 o are also simplyreferred to as “boundary lines 50 i and 50 o of the image pickup rangeRr.” The boundary line 50 i is parallel to the longitudinal axis A, butan extending direction of the boundary line 50 i is not limited thereto.The image pickup range Rr partially overlaps with the image pickup rangeRre (see the shaded part). The camera 11R picks up an image of a subjectpositioned in the image pickup range Rr, and outputs the obtained imagedata to the ECU 10 as right rear image data.

In FIG. 2 , for ease of viewing of the drawings, the own vehicle V andeach of the image pickup ranges Rre, Rl, and Rr are drawn in scalesdifferent from each other.

The description is continued referring back to FIG. 1 . The displaydevice 12 includes a display 12 a (display screen) provided at aposition viewable by a driver. The display device 12 may be, forexample, an inner mirror (rear-view mirror), or a display deviceincluding a navigation system. In the former case, a mirror part of theinner mirror is used as the display 12 a, and in the latter case, atouch panel is used as the display 12 a.

The ECU 10 acquires each of the rear image data, the left rear imagedata, and the right rear image data output from the cameras 11Re, 11L,and 11R, respectively, every time a predetermined time period T elapses.Then, the ECU 10 generates a panorama-format combined image (details aredescribed later) based on those acquired pieces of image data, anddisplays the combined image on the display 12 a of the display device12.

(Details of Operation)

Next, details of the operation of the ECU 10 are described. The CPU ofthe ECU 10 is configured to repeatedly execute, during a period in whichan ignition switch is turned on, a routine illustrated in a flow chartof FIG. 3 every time the predetermined time period T elapses.

When a predetermined timing arrives, the CPU starts the processing fromStep 300 of FIG. 3 , and sequentially executes process steps of fromStep 310 to Step 370.

In Step 310, the CPU acquires each of the rear image data, the left rearimage data, and the right rear image data output from the cameras 11Re,11L, and 11R, respectively.

In Step 320, the CPU analyzes each piece of image data acquired in Step310 to detect an object, and calculates a relative relationship betweenthe own vehicle V and the object. In this case, the “relativerelationship between the own vehicle V and the object” includes adistance from the own vehicle V to the object, a direction and arelative speed of the object with respect to the own vehicle V, and thelike.

Strictly speaking, the “distance from the own vehicle V to the object”means a distance (shortest distance) from any reference position(typically, rear-end center portion) of the rear end portion of the ownvehicle V to the object. That is, the CPU calculates a distance to theobject detected from each piece of image data (in other words, adistance to the object at the time when the position of each of thecameras 11Re, 11L, and 11R is set as a reference), converts thisdistance into a distance from the reference position of the own vehicleV, and calculates the converted distance as the “distance from the ownvehicle V to the object.” Further, strictly speaking, the “direction ofthe object with respect to the own vehicle V” means a direction of theobject with respect to the reference position of the own vehicle V. Thatis, the CPU calculates a direction of the object detected from eachpiece of image data (in other words, a direction of the object at thetime when the position of each of the cameras 11Re, 11L, and 11R is setas a reference), converts this direction into a direction with respectto the reference position of the own vehicle V, and calculates theconverted direction as the “direction of the object with respect to theown vehicle V.” In this manner, a position (distance and direction) ofthe object with respect to (the reference position of) the own vehicle Vmay be accurately calculated.

In addition, the CPU analyzes each piece of image data so as to detectthe dividing line. The dividing line is a line marked on a road in orderto separate the traffic of the vehicle in each direction. The CPUcalculates a position and a shape of a lane based on the detecteddividing line. The lane is defined as a region between two adjacentdividing lines extending on a roadway. The camera 11Re can detect atleast dividing lines forming an own lane L, a left lane Ll, and a rightlane Lr. In this case, the own lane L is a lane on which the own vehicleV is positioned. The left lane Ll is a lane adjacent to the own lane Lon its left side, and the right lane Lr is a lane adjacent to the ownlane L on its right side. The camera 11L can detect at least dividinglines forming the left lane Ll. The camera 11R can detect at leastdividing lines forming the right lane Lr.

In the example of FIG. 4 , the image pickup range Rre includes othervehicles Vre, Vl, and Vr2 as the objects, and also includes fourdividing lines d1 to d4. Accordingly, the CPU analyzes the rear imagedata so as to detect the other vehicles Vre, Vl, and Vr2, and calculatesthe relative relationship between the own vehicle V and each of theother vehicles Vre, Vl, and Vr2. In addition, the CPU analyzes the rearimage data so as to detect the dividing lines d1 to d4, and calculatesthe position and the shape of each of the own lane L, the left lane LI,and the right lane Lr.

The image pickup range Rl includes the another vehicle Vl as the object,and also includes the two dividing lines d1 and d2. Accordingly, the CPUanalyzes the left rear image data so as to detect the another vehicleVl, and calculates the relative relationship between the own vehicle Vand the another vehicle Vl. In addition, the CPU analyzes the left rearimage data so as to detect the dividing lines d1 and d2, and calculatesthe position and the shape of the left lane Ll. The object (in thisexample, the another vehicle Vl) and the dividing line (in this example,the dividing lines d1 and d2) which are positioned in an overlappingpart between the image pickup range Rre and the image pickup range Rlare detected separately from each of the rear image data and the leftrear image data.

The image pickup range Rr includes other vehicles Vr1 and Vr2 as theobjects, and also includes the two dividing lines d3 and d4.Accordingly, the CPU analyzes the right rear image data so as to detectthe other vehicles Vr1 and Vr2, and calculates the relative relationshipbetween the own vehicle V and the other vehicles Vr1 and Vr2. Inaddition, the CPU analyzes the right rear image data so as to detect thedividing lines d3 and d4, and calculates the position and the shape ofthe right lane Lr. The object (in this example, another vehicle Vr2) andthe dividing line (in this example, the dividing lines d3 and d4) whichare positioned in an overlapping part between the image pickup range Rreand the image pickup range Rr are detected separately from each of therear image data and the right rear image data.

In Step 330, the CPU determines, based on the computation resultsobtained in Step 320, on which lane each of the detected objects ispositioned. Then, a target object of each lane (own lane L, left laneLl, and right lane Lr) is identified. In this case, a “target object ofa lane” means, among objects positioned on the rear side with respect tothe rear end portion of the own vehicle V in this lane, an object havingthe shortest distance from the own vehicle V. The “objects positioned onthe rear side with respect to the rear end portion of the own vehicle V”include an “object having at least a part positioned on the rear sidewith respect to the rear end portion of the own vehicle V.” In thefollowing, the description of “having the shortest distance from the ownvehicle V” is also simply expressed as “closest to the own vehicle V.”Meanwhile, when the object positioned on the rear side with respect tothe rear end portion of the own vehicle V is not detected on a certainlane, the CPU determines that the target object of this lane is absent.

In the example of FIG. 4 , the CPU determines, based on the computationresults obtained by analyzing the rear image data, that the anothervehicle Vre is positioned on the own lane L, the another vehicle Vl ispositioned on the left lane Ll, and the another vehicle Vr2 ispositioned on the right lane Lr. Further, the CPU determines, based onthe computation results obtained by analyzing the left rear image data,that the another vehicle Vl is positioned on the left lane Ll. Moreover,the CPU determines, based on the computation results obtained byanalyzing the right rear image data, that both of the other vehicles Vr1and Vr2 are positioned on the right lane Lr.

The object on the own lane L is included only in the rear image data.Accordingly, the CPU identifies, among the objects determined as beingpositioned on the own lane L, the object closest to the own vehicle V(in the example of FIG. 4 , the another vehicle Vre) as the targetobject of the own lane L (rear target object).

The object on the left lane Ll may be included in both of the rear imagedata and the left rear image data. Accordingly, the CPU identifies,among “objects Ol(re) determined as being positioned on the left lane Llbased on the rear image data (in the example of FIG. 4 , the anothervehicle Vl)” and “objects Ol(l) determined as being positioned on theleft lane Ll based on the left rear image data (in the example of FIG. 4, the another vehicle Vl),” the object closest to the own vehicle V asthe target object of the left lane Ll. When the distance from the ownvehicle V to the object OI(re) and the distance from the own vehicle Vto the object Ol(l) are equal to each other, the CPU identifies any oneof the objects (in the example of FIG. 4 , for example, the anothervehicle Vl detected from the rear image data) as the target object ofthe left lane Ll (left rear target object). The same holds true also ina case in which the target object of the right lane Lr is identified.

The object on the right lane Lr may be included in both of the rearimage data and the right rear image data. Accordingly, the CPUidentifies, among “objects Or(re) determined as being positioned on theright lane Lr based on the rear image data (in the example of FIG. 4 ,the another vehicle Vr2)” and “objects Or(r) determined as beingpositioned on the right lane Lr based on the right rear image data (inthe example of FIG. 4 , the other vehicles Vr1 and Vr2),” the objectclosest to the own vehicle V (in the example of FIG. 4 , the anothervehicle Vr1) as the target object (right rear target object) of theright lane Lr.

In Step 340, the CPU determines, based on the position of the targetobject of each lane identified in Step 330, a zone in which each targetobject is positioned. Specifically, as illustrated in FIG. 4 , the CPUsets a zone Z1, a zone Z2, and a zone Z3 on the rear side of the ownvehicle V. The zone Z1 is a region between a straight line 21 (firststraight line) and a straight line 22 (second straight line). Thestraight line 21 passes through the rear end portion of the own vehicleV, and extends in a vehicle width direction (direction orthogonal to thelongitudinal axis A). The straight line 22 is separated away from thestraight line 21 rearward by a predetermined distance D1, and isparallel to the straight line 21. The zone Z2 is a region between thestraight line 22 and a straight line 23 (third straight line). Thestraight line 23 is separated away from the straight line 21 rearward bya predetermined distance D2, and is parallel to the straight line 21.The zone Z3 is a region on the rear side with respect to the straightline 23. The zone Z1 is a region relatively close to the own vehicle V,and the zone Z3 is a region relatively far from the own vehicle V. Thezone Z2 is an intermediate region between the zone Z1 and the zone Z3.

The distance D1 may be set in advance through experiments or simulationsso that, when the own vehicle V travels at relatively middle-levelvehicle speed or at high speed, the number of objects positioned in thezone Z1 is relatively small, and so that, when the own vehicle V is in astop state or travels at relatively low speed, the number of objectspositioned in the zone Z1 is relatively large. In the at least oneembodiment, D1=15 m is set. The distance D2 may be set in advancethrough experiments or simulations so that, when the own vehicle Vtravels at relatively high speed, the number of objects positioned inthe zone Z2 is relatively small, and so that, when the own vehicle Vtravels at relatively middle-level vehicle speed, the number of objectspositioned in the zone Z2 is relatively large. In the at least oneembodiment, D2=30 m is set. The zone Z3 is a region in which, when theown vehicle V travels at relatively high speed, the number of objectstends to be relatively large.

In the example of FIG. 4 , the CPU determines that the another vehicleVre being the target object of the own lane L is positioned in the zoneZ3, determines that the another vehicle Vl being the target object ofthe left lane Ll is positioned in the zone Z2, and determines that theanother vehicle Vr1 being the target object of the right lane Lr ispositioned in the zone Z1.

In Step 350, the CPU determines each of an azimuth angle θl and anazimuth angle θr with reference to matrices M1 and M2 to be describedlater. In the following, the azimuth angles θl and θr are firstdescribed, and then the process step of Step 350 is specificallydescribed.

Data ranges in the horizontal direction of the rear image data, the leftrear image data, and the right rear image data correspond to thehorizontal angles of view of the cameras 11Re, 11L, and 11R (in otherwords, the image pickup ranges Rre, Rl, and Rr), respectively. Asillustrated in FIG. 2 , the image pickup ranges Rre and RI partiallyoverlap with each other (in the horizontal direction), and the imagepickup ranges Rre and Rr partially overlap with each other (in thehorizontal direction). Accordingly, when those pieces of image data aredirectly used to generate a combined image, a range in which an objectand/or a dividing line is displayed doubly is increased in the combinedimage, and a seam is less likely to be smooth. In view of the above, theECU 10 performs trimming of an end portion in the horizontal directionof each of those pieces of image data as required so that the seam ofthe combined image becomes smoother. In the following, the image pickuprange Rre corresponding to the trimmed rear image data is referred to as“effective range Rrea (rear effective range),” the image pickup range Rlcorresponding to the trimmed left rear image data is referred to as“effective range Rla (left rear effective range),” and the image pickuprange Rr corresponding to the trimmed right rear image data is referredto as “effective range Rra (right rear effective range).” The effectiveranges Rrea, Rla, and Rra are image pickup ranges to be used forgenerating the combined image in the image pickup ranges Rre, Rl, andRr, respectively.

FIG. 5A and FIG. 5B are each a view for illustrating an example of theeffective ranges Rrea, Rla, and Rra. A boundary line 30 la and aboundary line 30 ra are, among boundary lines defining the horizontalangle of view of the effective range Rrea, a boundary line on the leftside of the own vehicle V (rear left-side boundary line) and a boundaryline on the right side of the own vehicle V (rear right-side boundaryline), respectively. In the following, the boundary lines 30 la and 30ra are also simply referred to as “boundary lines 30 la and 30 ra of theeffective range Rrea.” When a direction of the optical axis Are of thecamera 11Re is set as a rear reference direction (0°), an angle of theboundary line 30 la from the rear reference direction is defined as“azimuth angle θl (rear left-side azimuth angle),” and an angle of theboundary line 30 ra from the rear reference direction is defined as“azimuth angle θr (rear right-side azimuth angle).” It is assumed thatthe azimuth angle on the left side of the own vehicle V (right side ofthe drawing sheet) with respect to the rear reference direction has apositive value, and the azimuth angle on the right side of the ownvehicle V (left side of the drawing sheet) with respect to the rearreference direction has a negative value. The ECU 10 is configured toallow the effective range Rrea to be changed by changing the magnitudesof the azimuth angles θl and θr independently of each other within therange of the horizontal angle of view of the camera 11Re.

Specifically, the ECU 10 can change the azimuth angle θl between anangle θn (>0) (first rear left-side angle) and an angle θw (>0) (secondrear left-side angle), and can change the azimuth angle θr between anangle −θn (<0) (first rear right-side angle) and an angle −θw (<0)(second rear right-side angle). The angle θw is larger than the angleθn, and the magnitude of the angle −θw is larger than the magnitude ofthe angle −θn. FIG. 5A shows an example of a case in which the azimuthangles θl and θr are changed to the angles θn and −θn, respectively, andFIG. 5B shows an example of a case in which the azimuth angles θl and θrare changed to the angles θw and −θw, respectively. As illustrated inFIG. 5B, when the azimuth angle θl is the angle θw, the boundary line 30la of the effective range Rrea matches the boundary line 30 l of theimage pickup range Rre, and when the azimuth angle θr is the angle −θw,the boundary line 30 ra of the effective range Rrea matches the boundaryline 30 r of the image pickup range Rre. That is, the angle θw is alsoan azimuth angle (angle from the rear reference direction) of theboundary line 30 l of the image pickup range Rre, and the angle −θw isalso an azimuth angle (angle from the rear reference direction) of theboundary line 30 r of the image pickup range Rre. This means that, “whenthe azimuth angle θl is changed to the angle θw, the trimming of theleft-side end portion of the rear image data is not performed, and whenthe azimuth angle θr is changed to the angle −θw, the trimming of theright-side end portion of the rear image data is not performed.” As isclear from the above description, all of the angles θn, −θn, θw, and −θware angles within the range of the horizontal angle of view of thecamera 11Re.

The present disclosure is not limited to the configuration in which themagnitudes of the angle θw and the angle −θw match the magnitudes of theazimuth angles of the boundary lines 30 l and 30 r of the image pickuprange Rre, respectively, and embodiments of the present disclosure mayemploy a configuration in which the magnitudes of the angle θw and theangle −θw are smaller than the magnitudes of the azimuth angles of theboundary lines 30 l and 30 r, respectively. That is, in the case ofθl=θw or θr=−θw, the boundary line 30 la or 30 ra of the effective rangeRrea is not required to match the boundary line 30 l or 30 r of theimage pickup range Rre.

A boundary line 40 ia and a boundary line 40 oa are, among boundarylines defining the horizontal angle of view of the effective range Rla,a boundary line on the inner side in the vehicle width direction of theown vehicle V (left rear inner-side boundary line) and a boundary lineon the outer side in the vehicle width direction, respectively. In thefollowing, the boundary lines 40 ia and 40 oa are also simply referredto as “boundary lines 40 ia and 40 oa of the effective range Rla.” Whena direction of the optical axis Al of the camera 11L is set as a leftrear reference direction (0°), an angle of the boundary line 40 ia fromthe left rear reference direction is defined as “azimuth angle θli (leftrear inner-side azimuth angle).” It is assumed that the azimuth angle onthe outer side in the vehicle width direction (right side of the drawingsheet) with respect to the left rear reference direction has a positivevalue, and the azimuth angle on the inner side in the vehicle widthdirection (left side of the drawing sheet) with respect to the left rearreference direction has a negative value. The ECU 10 is configured toallow the effective range Rla to be changed by changing the magnitude ofthe azimuth angle θli within the range of the horizontal angle of viewof the camera 11L. The boundary line 40 oa of the effective range Rlamatches the boundary line 40 o of the image pickup range Rl (see FIG. 5Aand FIG. 5B). This means that “the trimming of the left-side end portionof the left rear image data is not performed.” That is, the ECU 10 isconfigured to perform only the trimming of the right-side end portion asrequired when generating the combined image.

Specifically, the ECU 10 can change the azimuth angle θli between anangle −θ1 (<0) (first left rear inner-side angle) and an angle −θ2 (<0)(second left rear inner-side angle). The magnitude of the angle −θ2 islarger than the magnitude of the angle −θ1. FIG. 5A shows an example ofa case in which the azimuth angle θli is changed to the angle −θ2, andFIG. 5B shows an example of a case in which the azimuth angle θli ischanged to the angle −θ1. As illustrated in FIG. 5A, when the azimuthangle θli is the angle −θ2, the boundary line 40 ia of the effectiverange Rla matches the boundary line 40 i of the image pickup range Rl.That is, the angle −θ2 is also the azimuth angle (angle from the leftrear reference direction) of the boundary line 40 i of the image pickuprange Rl. This means that, “when the azimuth angle θli is changed to theangle −θ2, the trimming of the right-side end portion of the left rearimage data is not performed (that is, the effective range Rla matchesthe image pickup range Rl).” As is clear from the above description,both of the angles −θ1 and −θ2 are angles within the range of thehorizontal angle of view of the camera 11L.

The present disclosure is not limited to the configuration in which themagnitude of the angle −θ2 matches the magnitude of the azimuth angle ofthe boundary line 40 i of the image pickup range Rl, and embodiments ofthe present disclosure may employ a configuration in which the magnitudeof the angle −θ2 is smaller than the magnitude of the azimuth angle ofthe boundary line 40 i. That is, in the case of θli=−θ2, the boundaryline 40 ia of the effective range Rla is not required to match theboundary line 40 i of the image pickup range Rl.

A boundary line 50 ia and a boundary line 50 oa are, among boundarylines defining the horizontal angle of view of the effective range Rra,a boundary line on the inner side in the vehicle width direction of theown vehicle V (right rear inner-side boundary line) and a boundary lineon the outer side in the vehicle width direction, respectively. In thefollowing, the boundary lines 50 ia and 50 oa are also simply referredto as “boundary lines 50 ia and 50 oa of the effective range Rra.” Whena direction of the optical axis Ar of the camera 11R is set as a rightrear reference direction (0°), an angle of the boundary line 50 ia fromthe right rear reference direction is defined as “azimuth angle θri(right rear inner-side azimuth angle).” It is assumed that the azimuthangle on the inner side in the vehicle width direction (right side ofthe drawing sheet) with respect to the right rear reference directionhas a positive value, and the azimuth angle on the outer side in thevehicle width direction (left side of the drawing sheet) with respect tothe right rear reference direction has a negative value. The ECU 10 isconfigured to allow the effective range Rra to be changed by changingthe magnitude of the azimuth angle θri within the range of thehorizontal angle of view of the camera 11R. The boundary line 50 oa ofthe effective range Rra matches the boundary line 50 o of the imagepickup range Rr (see FIG. 5A and FIG. 5B). This means that “the trimmingof the right-side end portion of the right rear image data is notperformed.” That is, the ECU 10 is configured to perform only thetrimming of the left-side end portion as required when generating thecombined image.

Specifically, the ECU 10 can change the azimuth angle θri between anangle θl (>0) (first right rear inner-side angle) and an angle θ2 (>0)(second right rear inner-side angle). The angle θ2 is larger than theangle θl. FIG. 5A shows an example of a case in which the azimuth angleθri is changed to the angle θ2, and FIG. 5B shows an example of a casein which the azimuth angle θri is changed to the angle θl. Asillustrated in FIG. 5A, when the azimuth angle θri is the angle θ2, theboundary line 50 ia of the effective range Rra matches the boundary line50 i of the image pickup range Rr. That is, the angle θ2 is also theazimuth angle (angle from the right rear reference direction) of theboundary line 50 i of the image pickup range Rr. This means that, “whenthe azimuth angle θri is changed to the angle θ2, the trimming of theleft-side end portion of the right rear image data is not performed.” Asis clear from the above description, both of the angles θ1 and θ2 areangles within the range of the horizontal angle of view of the camera11R.

The present disclosure is not limited to the configuration in which theangle θ2 matches the azimuth angle of the boundary lines 50 i of theimage pickup range Rr, and embodiments of the present disclosure mayemploy a configuration in which the angle θ2 is smaller than the azimuthangle of the boundary line 50 i. That is, in the case of θri=θ2, theboundary line 50 ia of the effective range Rra is not required to matchthe boundary line 50 i of the image pickup range Rr.

The ECU 10 is configured to set a projection surface Sp on the rear sideof the own vehicle V and set a viewpoint Pv on the own vehicle V, and togenerate, as the combined image, an image obtained by projectingsubjects (objects and dividing lines) present in the effective rangesRrea, Rla, and Rra onto the projection surface Sp through use of theviewpoint Pv as a reference. In this case, the projection surface Sp isan imaginary surface set so as to be orthogonal to a road surface (treadsurface of the own vehicle V) at a position separated away from thestraight line 21 (in other words, from the rear end portion of the ownvehicle V) rearward by a predetermined distance D3 (projectiondistance). In some embodiments, the projection surface Sp is set withinthe zone Z2. In some embodiments “D1<D3<D2” is satisfied. In moredetail, In some embodiments the projection surface Sp is set in a regionof the zone Z2 on a front side with respect to a center position in thedirection of the longitudinal axis A. In some embodiments“D1<D3<(D1+D2)/2” be satisfied. In the at least one embodiment, D3=20 mis set (that is, D3 is a fixed value), and “D1<D3<(D1+D2)/2” issatisfied. Depending on the values of D1 and D2, the projection surfaceSp may be set in another zone (zone Z1 or zone Z3).

The viewpoint Pv is set at an intersection between the extension line ofthe boundary line 40 oa and the extension line of the boundary line 50oa. The position of the viewpoint Pv does not change due to the trimmingof the left rear image data and/or the right rear image data. Theposition of the viewpoint Pv is not limited to a position on the ownvehicle V. Depending on the extending directions of the boundary line 40oa and the boundary line 50 oa, in some cases, the viewpoint Pv ispositioned not on the own vehicle V but on the front side of the ownvehicle V.

As described above, the distance D3 is a fixed value, and the extendingdirections of the boundary lines 40 oa and 50 oa do not changeregardless of whether or not the trimming is performed. Thus, a relativeposition of a point P21 (intersection between the boundary line 40 oaand the projection surface Sp in plan view of the own vehicle V) withrespect to the own vehicle V and a relative position of a point P22(intersection between the boundary line 50 oa and the projection surfaceSp in plan view of the own vehicle V) with respect to the own vehicle Vdo not change. In the following, the “plan view of the own vehicle V” isalso simply referred to as “plan view.”

As illustrated in FIG. 5A, the ECU 10 is configured to change, whenchanging the azimuth angle θl to the angle θn, the azimuth angle θli tothe angle −θ2. The values of the angle θn and the angle −θ2 are set inadvance so that a point Pl (left-side intersection) being anintersection between the boundary line 30 la of the effective range Rreaand the boundary line 40 ia of the effective range Rla is positioned onthe projection surface Sp in plan view. In the at least one embodiment,the angle −θ2 is also the azimuth angle of the boundary line 40 i of theimage pickup range Rl. Accordingly, the position of the intersectionbetween the boundary line 40 ia (40 i) and the projection surface Sp inplan view may be uniquely determined based on the mounting position andthe performance (typically, the horizontal angle of view) of the camera11L. The angle θn is set in advance to such a value that the boundaryline 30 la passes through the position of the intersection between theboundary line 40 ia and the projection surface Sp determined asdescribed above. With this configuration, in the case of the azimuthangle θl=θn, the point Pl is positioned on the projection surface Sp inplan view. In the following, the point Pl in the case of the azimuthangle θl=θn is also particularly referred to as “point Pln.”

Similarly, the ECU 10 is configured to change, when changing the azimuthangle θr to the angle −θn, the azimuth angle θri to the angle θ2. Thevalues of the angle −θn and the angle θ2 are set in advance so that apoint Pr (right-side intersection) being an intersection between theboundary line 30 ra of the effective range Rrea and the boundary line 50ia of the effective range Rra is positioned on the projection surface Spin plan view. In the at least one embodiment, the angle θ2 is also theazimuth angle of the boundary line 50 i of the image pickup range Rr.Accordingly, the position of the intersection between the boundary line50 ia (50 i) and the projection surface Sp in plan view may be uniquelydetermined based on the mounting position and the performance(typically, the horizontal angle of view) of the camera 11R. The angle−θn is set in advance to such a value that the boundary line 30 rapasses through the position of the intersection between the boundaryline 50 ia and the projection surface Sp determined as described above.With this configuration, in the case of the azimuth angle θr=−θn, thepoint Pr is positioned on the projection surface Sp in plan view. In thefollowing, the point Pr in the case of the azimuth angle θr=−θn is alsoparticularly referred to as “point Prn.”

As illustrated in FIG. 5B, the ECU 10 is configured to change, whenchanging the azimuth angle θl to the angle θw, the azimuth angle θli tothe angle −θ1. The values of the angle θw and the angle −θ1 are set inadvance so that the point Pl is positioned on the projection surface Spin plan view. In the at least one embodiment, the angle θw is also theazimuth angle of the boundary line 30 l of the image pickup range Rre.Accordingly, the position of the intersection between the boundary line30 la (30 l) and the projection surface Sp in plan view may be uniquelydetermined based on the mounting position and the performance(typically, the horizontal angle of view) of the camera 11Re. The angle−θ1 is set in advance to such a value that the boundary line 40 iapasses through the position of the intersection between the boundaryline 30 la and the projection surface Sp determined as described above.With this configuration, in the case of the azimuth angle θ1=0w, thepoint PI is positioned on the projection surface Sp in plan view. In thefollowing, the point Pl in the case of the azimuth angle θ1=0w is alsoparticularly referred to as “point Plw.”

Similarly, the ECU 10 is configured to change, when changing the azimuthangle θr to the angle −θw, the azimuth angle θri to the angle θl. Thevalues of the angle −θw and the angle θl are set in advance so that thepoint Pr is positioned on the projection surface Sp in plan view. In theat least one embodiment, the angle −θw is also the azimuth angle of theboundary line 30 r of the image pickup range Rre. Accordingly, theposition of the intersection between the boundary line 30 ra (30 r) andthe projection surface Sp in plan view may be uniquely determined basedon the mounting position and the performance (typically, the horizontalangle of view) of the camera 11Re. The angle θl is set in advance tosuch a value that the boundary line 50 ia passes through the position ofthe intersection between the boundary line 30 ra and the projectionsurface Sp determined as described above. With this configuration, inthe case of the azimuth angle θr=−θw, the point Pr is positioned on theprojection surface Sp in plan view. In the following, the point Pr inthe case of the azimuth angle θr=−θw is also particularly referred to as“point Prw.”

The point Pl is positioned at a seam (left-side seam) between thetrimmed rear image data and the trimmed left rear image data (in otherwords, between the image data corresponding to the effective range Rreain the rear image data and the image data corresponding to the effectiverange Rla in the left rear image data) in the combined image. Further,the point Pr is positioned at a seam (right-side seam) between thetrimmed rear image data and the trimmed right rear image data (in otherwords, between the image data corresponding to the effective range Rreain the rear image data and the image data corresponding to the effectiverange Rra in the right rear image data) in the combined image. The ECU10 is configured to allow the position (point Pl) of the left-side seamin the combined image to be changed between the point Pln and the pointPlw by changing the azimuth angle θl between the angle θn and the angleθw. Further, the ECU 10 is configured to allow the position (point Pr)of the right-side seam in the combined image to be changed between thepoint Prn and the point Prw by changing the azimuth angle θr between theangle −θn and the angle −θw.

In a region between the point Pl and the point Pr in the projectionsurface Sp, a subject present in the effective range Rrea is projected.In a region between the point Pl and the point P21 in the projectionsurface Sp, a subject present in the effective range Rla is projected.In a region between the point Pr and the point P22 in the projectionsurface Sp, a subject present in the effective range Rra is projected.

The positions of the point Pln and the point Prn are described in moredetail. As illustrated in FIG. 5A, a point P11 (first intersection) isan intersection between the projection surface Sp and a left imaginaryline extending rearward from a left rear corner portion Cl of the ownvehicle V so as to be parallel to the longitudinal axis A (see FIG. 2 ),and a point P12 (second intersection) is an intersection between theprojection surface Sp and a right imaginary line extending rearward froma right rear corner portion Cr of the own vehicle V so as to be parallelto the longitudinal axis A. In the at least one embodiment, in planview, the point Pln and the point Prn are positioned on the point P11and the point P12, respectively. The present disclosure is not limitedto the configuration in which the point Pln is positioned on the pointP11, and embodiments of the present disclosure may employ aconfiguration in which the point Pln is positioned in the vicinity ofthe point P11 based on the values of the angle θn and the angle −θ2.Similarly, the present disclosure is not limited to the configuration inwhich the point Prn is positioned on the point P12, and embodiments ofthe present disclosure may employ a configuration in which the point Prnis positioned in the vicinity of the point P12 based on the values ofthe angle −θn and the angle θ2.

The positions of the point Plw and the point Prw are described in moredetail. As illustrated in FIG. 5B, the point Plw is positioned at aposition separated away from the point P11 outward in the vehicle widthdirection, and a separation distance of the point Plw is sufficientlylarger than the lane width of the left lane Ll. Further, the point Prwis positioned at a position separated away from the point P12 outward inthe vehicle width direction, and a separation distance of the point Prwis sufficiently larger than the lane width of the right lane Lr. Theseparation distance of the point Plw from the point P11 may be equal tothe lane width of the left lane Ll, and the separation distance of thepoint Prw from the point P12 may be equal to the lane width of the rightlane Lr. That is, the point Plw and the point Prw may be set so as to bepositioned at positions separated away from the point P11 and the pointP12, respectively, outward in the vehicle width direction bypredetermined distances equal to or larger than a general lane width.

It is here assumed that, when each of the lanes L, LI, and Lr has astraight line shape, the own vehicle V travels along the extendingdirection of the own lane L (the longitudinal axis A is parallel to theextending direction). In this case, the point Pln is positioned slightlyon the inner side in the vehicle width direction with respect to thedividing line d2 (left-side dividing line forming the own lane L), andthe point Prn is positioned slightly on the inner side in the vehiclewidth direction with respect to the dividing line d3 (right-sidedividing line forming the own lane L) (see FIG. 5A). Further, the pointPlw is positioned on the outer side in the vehicle width direction withrespect to the dividing line d1 (left-side dividing line forming theleft lane Ll), and the point Prw is positioned on the outer side in thevehicle width direction with respect to the dividing line d4 (right-sidedividing line forming the right lane Lr) (see FIG. 5B). Moreover, theprojection surface Sp on which the point Pl (Pln or Plw) and the pointPr (Prn or Prw) are positioned is positioned in the zone Z2, and“D1<D3<(D1+D2)/2” is satisfied.

At this time, a position of each of the zones Z1 to Z3 at which a blindspot region Rb and/or an overlapping region Rov (both regions aredescribed later) is formed has a feature corresponding to the value ofθl (or |θr|) (that is, θn or θw). Now, the feature is specificallydescribed. The blind spot region Rb is a region between the effectiveranges Rrea and Rla and a region between the effective ranges Rrea andRra. A subject present in the blind spot region Rb is not displayed inthe combined image. The overlapping region Rov is a region in which theeffective ranges Rrea and Rla partially overlap with each other and aregion in which the effective ranges Rrea and Rra partially overlap witheach other. A subject present in the overlapping region Rov is displayeddoubly in the combined image. In the at least one embodiment, the pointPl and the point Pr are positioned on the projection surface Sp.Accordingly, the blind spot region Rb is formed on the front side of theprojection surface Sp (in other words, on the front side of the point Pland the point Pr), and the overlapping region Rov is formed on the rearside of the projection surface Sp (in other words, on the rear side ofthe point Pl and the point Pr) (see FIG. 5A and FIG. 5B).

First, the case of θl=θn or θr=−θn is described with reference to FIG.5A. The words in the parentheses are features in the case of θr=−θn. Inthe zone Z1, the following features are obtained.

(Feature 1n) The blind spot region Rb is formed in a part of the ownlane L excluding a center portion and a left end portion (right endportion) thereof.

(Feature 2n) No blind spot region Rb is formed in the left lane Ll(right lane Lr).

(Feature 3n) No overlapping region Rov is formed in the own lane L andthe left lane Ll (right lane Lr).

In the zone Z2, the following features are obtained.

(Feature 4n) The blind spot region Rb is slightly formed in a part ofthe own lane L on the front side of the point Pln (point Prn) excludingthe center portion and the left end portion (right end portion) thereof.

(Feature 5n) No blind spot region Rb is formed in the left lane Ll(right lane Lr).

(Feature 6n) The overlapping region Rov is slightly formed at the leftend portion (right end portion) of the own lane L on the rear side ofthe point Pln (point Prn).

(Feature 7n) Almost no overlapping region Rov is formed in the left laneLl (right lane Lr).

In the zone Z3, the following features are obtained.

(Feature 8n) No blind spot region Rb is formed in the own lane L and theleft lane Ll (right lane Lr).

(Feature 9n) The overlapping region Rov is slightly formed at the leftend portion (right end portion) of the own lane L.

(Feature 10n) The overlapping region Rov is formed at the right endportion (left end portion) of the left lane Ll (right lane Lr), and itsrange is increased as the overlapping region Rov expands toward the rearside.

Next, the case of θl=θw or θr=−θw is described with reference to FIG.5B. In the zone Z1, the following features are obtained.

(Feature 1w) The blind spot region Rb is slightly formed on the leftside (right side) of the front part of the own lane L.

(Feature 2w) The blind spot region Rb is formed so as to cross the leftlane Ll (right lane Lr) in a diagonally rearward left direction.

(Feature 3w) No overlapping region Rov is formed in the own lane L andthe left lane Ll (right lane Lr).

In the zone Z2, the following features are obtained.

(Feature 4w) No blind spot region Rb is formed in the own lane L.

(Feature 5w) No blind spot region Rb is formed in the left lane Ll(right lane Lr).

(Feature 6w) No overlapping region Rov is formed in the own lane L.

(Feature 7w) No overlapping region Rov is formed in the left lane Ll(right lane Lr).

In the zone Z3, the following features are obtained.

(Feature 8w) No blind spot region Rb is formed in the own lane L and theleft lane Ll (right lane Lr).

(Feature 9w) No overlapping region Rov is formed in the own lane L.

(Feature 10w) No overlapping region Rov is formed in the left lane Ll(right lane Lr).

Next, the process step of Step 350 is specifically described. The CPUdetermines (changes or maintains), through use of the determinationresults of Step 340, the azimuth angle θl based on a combination of“presence or absence of a target object of the own lane L and a zone inwhich this target object is positioned” and “presence or absence of atarget object of the left lane Ll and a zone in which this target objectis positioned.” FIG. 6A shows the matrix M1 representing thiscombination. Regarding the target object of the own lane L, “Z1(close),” “Z2 (intermediate),” “Z3 (far),” and “absent” in the matrix M1represent that this target object is “positioned in the zone Z1,” thistarget object is “positioned in the zone Z2,” this target object is“positioned in the zone Z3,” and “the target object is absent,”respectively. The same holds true also for the target object of the leftlane Ll. Further, the same holds true also for the target objects of theown lane L and the right lane Lr in the matrix M2 to be described later.In addition, “narrow” and “wide” in the matrix M1 mean that the azimuthangle θl is “changed to the angle θn” and the azimuth angle θl is“changed to the angle θw,” respectively. “No change” means that theazimuth angle θl is not changed and maintained to a value in a cycleimmediately before the current cycle. The CPU changes the azimuth angleθli to the above-mentioned angle (−θ2 or −θ1) in accordance with thechange of the azimuth angle θl.

In the example of FIG. 4 , the target object of the own lane L (anothervehicle Vre) is positioned in the zone Z3, and the target object of theleft lane Ll (another vehicle Vl) is positioned in the zone Z2. In thiscase, with reference to the matrix M1, the azimuth angle θl correspondsto “no change,” and hence the CPU does not change the azimuth angle θl.That is, for example, when θl=θw is set in the cycle immediately beforethe current cycle, the CPU maintains the azimuth angle θl at the angleθw also in the current cycle (see FIG. 5B).

Similarly, the CPU determines (changes or maintains), through use of thedetermination results of Step 340, the azimuth angle θr based on acombination of “presence or absence of a target object of the own lane Land a zone in which this target object is positioned” and “presence orabsence of a target object of the right lane Lr and a zone in which thistarget object is positioned.” FIG. 6B shows the matrix M2 representingthis combination. “Narrow” and “wide” in the matrix M2 mean that theazimuth angle θr is “changed to the angle −θn” and the azimuth angle θris “changed to the angle −θw,” respectively. “No change” means that theazimuth angle θr is not changed and maintained to a value in a cycleimmediately before the current cycle. The CPU changes the azimuth angleθri to the above-mentioned angle (θ2 or 01) in accordance with thechange of the azimuth angle θr.

In the example of FIG. 4 , the target object of the own lane L (anothervehicle Vre) is positioned in the zone Z3, and the target object of theright lane Lr (another vehicle Vr1) is positioned in the zone Z1. Inthis case, with reference to the matrix M2, the azimuth angle θrcorresponds to “narrow,” and hence the CPU changes the azimuth angle θrto the angle −θn (see FIG. 5A).

In Step 360, the CPU trims, based on the azimuth angles θl and θrdetermined in Step 350, the pieces of image data acquired in Step 310,and generates a panorama-format combined image by a well-known methodthrough use of the trimmed pieces of image data. In this manner, theposition of the left-side seam in the combined image becomes the pointPln when θl=θn is set (see FIG. 5A), and becomes the point Plw whenθl=θw is set (see FIG. 5B). Further, the position of the right-side seamin the combined image becomes the point Prn when θr=−θn is set (see FIG.5A), and becomes the point Prw when θr=−θw is set (see FIG. 5B).

In Step 370, the CPU displays the combined image generated in Step 360on the display 12 a of the display device 12. After that, the processproceeds to Step 395, and the CPU temporarily ends this routine.

Next, with reference to FIG. 11 to FIG. 13 , problems of image controlsystems serving as comparative examples are described. In the imagecontrol systems of FIG. 11 to FIG. 13 , the position of the seam of thecombined image cannot be changed. Like configurations as those of theimage control system 1 are denoted by like reference symbols, anddetailed description thereof is omitted.

In the example of FIG. 11 and FIG. 12 , an image control system servingas a first comparative example is mounted on a vehicle V1. This imagecontrol system includes cameras 11Ren, 11L, and 11R. The horizontalangle of view of the camera 11Ren is relatively narrow, and thehorizontal angle of view of each of the cameras 11L and 11R isrelatively wide. Accordingly, an image pickup range Rren has anelongated fan shape, and each of image pickup ranges Rl and Rr has awide fan shape. Each of a point P2 and a point P3 represents a positionof a seam in a combined image. The positions of the point P2 and thepoint P3 cannot be changed. The vehicle V1 travels on the own lane L.

As illustrated in FIG. 11 , the blind spot region Rb is formed betweenthe image pickup ranges Rren and Rl on the front side of the point P2.Similarly, the blind spot region Rb is formed between the image pickupranges Rren and Rr on the front side of the point P3. On the rear sideof the vehicle V1, a two-wheeled motor vehicle Mre and another vehicleVre travel on the own lane L. Almost all part of the two-wheeled motorvehicle Mre including its center part is positioned in the blind spotregion Rb, and only left and right end portions thereof are included inthe image pickup ranges Rl and Rre, respectively. Accordingly, thecenter part of the two-wheeled motor vehicle Mre is cut off from thecombined image, and only the left and right end portions are displayed.Meanwhile, a center part of the another vehicle Vre is positioned in theimage pickup range Rren, and left and right end portions thereof arepositioned in the blind spot regions Rb. Accordingly, the left and rightend portions of the another vehicle Vre are cut off from the combinedimage, and only the center part thereof is displayed.

Further, as illustrated in FIG. 12 , the image pickup ranges Rren and Rlpartially overlap with each other on the rear side of the point P2, andthus the overlapping region Rov is formed. Similarly, the image pickupranges Rren and Rr partially overlap with each other on the rear side ofthe point P3, and thus the overlapping region Rov is formed. On the rearside of the vehicle V1, the another vehicle Vl travels on the left laneLl. A part of the another vehicle Vl is positioned in the overlappingregion Rov. Accordingly, in the combined image, a part of the anothervehicle Vl is displayed doubly.

As described above, when the horizontal angle of view of the rear camerais relatively narrow, there are caused such problems that a part of theobject is cut off from the combined image and at least a part of theobject is displayed doubly in the combined image. In the following, theformer problem is also referred to as “cut-off phenomenon,” and thelatter problem is also referred to as “doubling phenomenon.”

In the example of FIG. 13 , an image control system serving as a secondcomparative example is mounted on a vehicle V2. This image controlsystem includes cameras 11Re, 11Ln, and 11Rn. The horizontal angle ofview of the camera 11Re is relatively wide, and the horizontal angle ofview of each of the cameras 11Ln and 11Rn is relatively narrow.

Accordingly, an image pickup range Rre has a wide fan shape, and each ofimage pickup ranges Rln and Rrn has a slightly elongated fan shape. Eachof a point P5 and a point P6 represents a position of a seam in acombined image. The positions of the point P5 and the point P6 cannot bechanged. The vehicle V2 travels on the own lane L.

As illustrated in FIG. 13 , the blind spot regions Rb are formed betweenthe image pickup ranges Rre and Rln and between the image pickup rangesRre and Rrn, on the front side of the point P5 and the point P6,respectively. On the rear side of the vehicle V2, the another vehicleVre travels on the own lane L, and the two-wheeled motor vehicle Mre andthe another vehicle Vl travel on the left lane Ll. The entiretwo-wheeled motor vehicle Mre is positioned in the blind spot region Rb.Accordingly, the two-wheeled motor vehicle Mre is not displayed in thecombined image (disappears in the combined image). Further, both of theanother vehicle Vre and the another vehicle Vl are present in the imagepickup range Rre, but light from the another vehicle Vl is blocked bythe another vehicle Vre and thus does not reach the camera 11Re. Thus,the camera 11Re cannot pick up the image of the another vehicle Vl, andas a result, the another vehicle Vl is not displayed in the combinedimage (is hidden behind another object in the combined image).

As described above, when the horizontal angle of view of the rear camerais relatively wide, there are caused such problems that the objectdisappears in the combined image and the object on one lane is hiddenbehind the object on another lane. In the following, the former problemis also referred to as “disappearing phenomenon,” and the latter problemis also referred to as “hiding phenomenon.”

Also in the examples of FIG. 11 and FIG. 12 , the disappearingphenomenon may occur. Further, also in the example of FIG. 13 , thecut-off phenomenon may occur (the doubling phenomenon may also occur,but the doubling phenomenon does not occur at least on the own lane L,the left lane Ll, and the right lane Lr, and hence this case is notconsidered in this specification). However, the hiding phenomenon isliable to occur when, in a case in which the horizontal angle of view ofthe rear camera is relatively wide, the object on the left lane Ll orthe right lane Lr is positioned diagonally rearward with respect to theobject on the own lane L. Accordingly, when the horizontal angle of viewof the rear camera is relatively narrow, the hiding phenomenon does notpossibly occur.

As described above, when the horizontal angle of view of the rear camerais relatively narrow, there are problems of cut-off, disappearing, anddoubling phenomena of the object in the combined image. Meanwhile, whenthe horizontal angle of view of the rear camera is relatively wide,there are problems of cut-off, disappearing, and hiding phenomena of theobject in the combined image.

In contrast, the image control system 1 is configured to allow theposition of the seam in the combined image to be changed by changing themagnitudes of the azimuth angles θl and θr between the angles θn and theangles θw. Accordingly, among the above-mentioned phenomena (cut-off,disappearing, doubling, and hiding phenomena), occurrence of at least aphenomenon having a possibility of most affecting the traveling of theown vehicle V can be suppressed. Now, with reference to the matrix M1,specific description is given through use of FIG. 7A to FIG. 10B (thematrix M2 has a configuration similar to that of the matrix M1, andhence detailed description thereof is omitted). In FIG. 7A to FIG. 10B,the target object is absent on the right lane Lr. In this case, withreference to the matrix M2, when the target object is absent on the ownlane L (see FIG. 7A and FIG. 7B), when this target object is positionedin the zone Z3 (see FIG. 8A and FIG. 8B), and when this target object ispositioned in the zone Z2 (see FIG. 9A and FIG. 9B), the CPU maintainsthe azimuth angle θr to the value in the cycle immediately before thecurrent cycle (“no change”). Thus, in each of FIG. 7A to FIG. 9B, theazimuth angle θr is shown assuming that the value of the azimuth angleθr at the time when the value in the cycle immediately before thecurrent cycle is maintained is equal to the value of the azimuth angleθl in each figure.

Meanwhile, with reference to the matrix M2, when the target object ofthe own lane L is positioned in the zone Z1 (see FIG. 10A and FIG. 10B),the CPU changes the azimuth angle θr to the angle −θw (“wide”).Accordingly, in FIG. 10A and FIG. 10B, the azimuth angle θr is shown sothat the azimuth angle θr becomes the angle −θw.

1. When Target Object is Absent on Own Lane L (FIG. 7A and FIG. 7B)

1-1. When Target Object of Left Lane Ll is Another Vehicle Vl1

Another vehicle Vl1 is positioned in the zone Z1. Accordingly, the CPUchanges the azimuth angle θl to the angle θn (“narrow”) with referenceto the matrix M1. As illustrated in FIG. 7A, in this case, the feature2n and the feature 3n are satisfied, and hence the another vehicle Vl1may be appropriately displayed in the combined image. In contrast, whenthe azimuth angle θl is changed to the angle θw, the feature 2w and thefeature 3w are satisfied (see FIG. 7B), and hence the another vehicleVl1 cannot be appropriately displayed in the combined image due to thecut-off phenomenon. Thus, in the case of Section 1-1, when the azimuthangle θl is changed to the angle θn, the target object of the left laneLl can be appropriately displayed in the combined image.

1-2. When Target Object of Left Lane Ll is Another Vehicle Vl2

Another vehicle Vl2 is positioned in the zone Z2. Accordingly, the CPUmaintains the azimuth angle θl to the value in the cycle immediatelybefore the current cycle with reference to the matrix M1 (“no change”).When the azimuth angle θl in the cycle immediately before the currentcycle is the angle θn, the CPU maintains θl=θn in the current cycle. Inthis case, as illustrated in FIG. 7A, the feature 5n and the feature 7nare satisfied, and hence the another vehicle Vl2 may be appropriatelydisplayed in the combined image. Meanwhile, when the azimuth angle θl inthe cycle immediately before the current cycle is the angle θw, the CPUmaintains θl=θw in the current cycle. In this case, as illustrated inFIG. 7B, the feature 5w and the feature 7w are satisfied, and hence theanother vehicle Vl2 may be appropriately displayed in the combinedimage. That is, in the case of Section 1-2, regardless of the angle (θnor θw) to which the azimuth angle θl is changed, the target object ofthe left lane Ll can be appropriately displayed in the combined image.

1-3. When Target Object of Left Lane Ll is Another Vehicle Vl3

Another vehicle Vl3 is positioned in the zone Z3. Accordingly, the CPUchanges the azimuth angle θl to the angle θw (“wide”) with reference tothe matrix M1. As illustrated in FIG. 7B, in this case, the feature 8wand the feature 10w are satisfied, and hence the another vehicle Vl3 maybe appropriately displayed in the combined image. In contrast, when theazimuth angle θl is changed to the angle θn, the feature 8n and thefeature 10n are satisfied (see FIG. 7A), and hence the another vehicleVl3 cannot be appropriately displayed in the combined image due to thedoubling phenomenon. Thus, in the case of Section 1-3, when the azimuthangle θl is changed to the angle θw, the target object of the left laneLl can be appropriately displayed in the combined image.

1-4. When Target Object is Absent on Left Lane Ll

In this case, the target object is not present on any of the own lane Lor the left lane Ll, and hence regardless of the angle to which theazimuth angle θl is changed, the above-mentioned phenomena (cut-off,disappearing, doubling, and hiding phenomena) do not possibly occur.

2. When Target Object of Own Lane L (Another Vehicle Vre) is Positionedin Zone Z3 (FIG. 8A and FIG. 8B)

2-1. When Target Object of Left Lane Ll is Another Vehicle Vl1

The another vehicle Vl1 is positioned in the zone Z1. Accordingly, theCPU changes the azimuth angle θl to the angle θn (“narrow”) withreference to the matrix M1. As illustrated in FIG. 8A, in this case, thefeature 8n and the feature 9n are satisfied for the another vehicle Vre,and hence the another vehicle Vre may be appropriately displayed in thecombined image. Further, the feature 2n and the feature 3n are satisfiedfor the another vehicle Vl1, and hence the another vehicle Vl1 may beappropriately displayed in the combined image.

In contrast, when the azimuth angle θl is changed to the angle θw (seeFIG. 8B), the feature 8w and the feature 9w are satisfied for theanother vehicle Vre, and hence the another vehicle Vre may beappropriately displayed in the combined image. Meanwhile, the feature 2wand the feature 3w are satisfied for the another vehicle Vl1, and hencethe another vehicle Vl1 cannot be appropriately displayed in thecombined image due to the cut-off phenomenon.

Thus, in the case of Section 2-1, when the azimuth angle θl is changedto the angle θn, the target objects of the own lane L and the left laneLl can be appropriately displayed in the combined image.

2-2. When Target Object of Left Lane Ll is Another Vehicle Vl2

The another vehicle Vl2 is positioned in the zone Z2. Accordingly, theCPU maintains the azimuth angle θl to the value in the cycle immediatelybefore the current cycle (“no change”) with reference to the matrix M1.When the azimuth angle θl in the cycle immediately before the currentcycle is the angle θn, the CPU maintains θl=θn in the current cycle. Inthis case, as illustrated in FIG. 8A, the feature 8n and the feature 9nare satisfied for the another vehicle Vre, and hence the another vehicleVre may be appropriately displayed in the combined image. Further, thefeature 5n and the feature 7n are satisfied for the another vehicle Vl2,and hence the another vehicle Vl2 may be appropriately displayed in thecombined image.

Meanwhile, when the azimuth angle θl in the cycle immediately before thecurrent cycle is the angle θw, the CPU maintains θl=θw in the currentcycle. In this case, as illustrated in FIG. 8B, the feature 8w and thefeature 9w are satisfied for the another vehicle Vre, and hence theanother vehicle Vre may be appropriately displayed in the combinedimage. Further, the feature 5w and the feature 7w are satisfied for theanother vehicle Vl2, and hence the another vehicle Vl2 may beappropriately displayed in the combined image.

Thus, in the case of Section 2-2, regardless of the angle (θn or θw) towhich the azimuth angle θl is changed, the target objects of the ownlane L and the left lane Ll can be appropriately displayed in thecombined image.

2-3. When Target Object of Left Lane Ll is Another Vehicle Vl3

The another vehicle Vl3 is positioned in the zone Z3. Accordingly, theCPU changes the azimuth angle θl to the angle θw (“wide”) with referenceto the matrix M1. As illustrated in FIG. 8B, in this case, the feature8w and the feature 9w are satisfied for the another vehicle Vre, andhence the another vehicle Vre may be appropriately displayed in thecombined image. Further, the feature 8w and the feature 10w aresatisfied for the another vehicle Vl3, and hence the another vehicle Vl3may be appropriately displayed in the combined image.

In contrast, when the azimuth angle θl is changed to the angle θn (seeFIG. 8A), the feature 8n and the feature 9n are satisfied for theanother vehicle Vre, and hence the another vehicle Vre may beappropriately displayed in the combined image. Meanwhile, the feature 8nand the feature 10n are satisfied for the another vehicle Vl3, and hencethe another vehicle Vl3 cannot be appropriately displayed in thecombined image due to the doubling phenomenon.

Thus, in the case of Section 2-3, when the azimuth angle θl is changedto the angle θw, the target objects of the own lane L and the left laneLl can be appropriately displayed in the combined image.

2-4. When Target Object is Absent on Left Lane Ll

In this case, the CPU maintains the azimuth angle θl to the value in thecycle immediately before the current cycle (“no change”) with referenceto the matrix M1. When the azimuth angle θl in the cycle immediatelybefore the current cycle is the angle θn, the CPU maintains θl=θn in thecurrent cycle. In this case, as illustrated in FIG. 8A, the feature 8nand the feature 9n are satisfied for the another vehicle Vre, and hencethe another vehicle Vre may be appropriately displayed in the combinedimage.

Meanwhile, when the azimuth angle θl in the cycle immediately before thecurrent cycle is the angle θw, the CPU maintains θl=θw in the currentcycle. In this case, as illustrated in FIG. 8B, the feature 8w and thefeature 9w are satisfied for the another vehicle Vre, and hence theanother vehicle Vre may be appropriately displayed in the combinedimage.

Thus, in the case of Section 2-4, regardless of the angle (θn or θw) towhich the azimuth angle θl is changed, the target object of the own laneL can be appropriately displayed in the combined image.

3. When Target Object of Own Lane L (Another Vehicle Vre) is Positionedin Zone Z2 (FIG. 9A and FIG. 9B)

3-1. When Target Object of Left Lane Ll is Another Vehicle Vl1

The another vehicle Vl1 is positioned in the zone Z1. Accordingly, theCPU changes the azimuth angle θl to the angle θn (“narrow”) withreference to the matrix M1. As illustrated in FIG. 9A, in this case, thefeature 4n and the feature 6n are satisfied for the another vehicle Vre,and hence the another vehicle Vre may be appropriately displayed in thecombined image. Further, the feature 2n and the feature 3n are satisfiedfor the another vehicle Vl1, and hence the another vehicle Vl1 may beappropriately displayed in the combined image.

In contrast, when the azimuth angle θl is changed to the angle θw (seeFIG. 9B), the feature 4w and the feature 6w are satisfied for theanother vehicle Vre, and hence the another vehicle Vre may beappropriately displayed in the combined image. Meanwhile, the feature 2wand the feature 3w are satisfied for the another vehicle Vl1, and hencethe another vehicle Vl1 cannot be appropriately displayed in thecombined image due to the cut-off phenomenon.

Thus, in the case of Section 3-1, when the azimuth angle θl is changedto the angle θn, the target objects of the own lane L and the left laneLl can be appropriately displayed in the combined image.

3-2. When Target Object of Left Lane Ll is Another Vehicle Vl2

The another vehicle Vl2 is positioned in the zone Z2. Accordingly, theCPU maintains the azimuth angle θl to the value in the cycle immediatelybefore the current cycle (“no change”) with reference to the matrix M1.When the azimuth angle θl in the cycle immediately before the currentcycle is the angle θn, the CPU maintains θl=θn in the current cycle. Inthis case, as illustrated in FIG. 9A, the feature 4n and the feature 6nare satisfied for the another vehicle Vre, and hence the another vehicleVre may be appropriately displayed in the combined image. Further, thefeature 5n and the feature 7n are satisfied for the another vehicle Vl2,and hence the another vehicle Vl2 may be appropriately displayed in thecombined image.

Meanwhile, when the azimuth angle θl in the cycle immediately before thecurrent cycle is the angle θw, the CPU maintains θl=θw in the currentcycle. In this case, as illustrated in FIG. 9B, the feature 4w and thefeature 6w are satisfied for the another vehicle Vre, and hence theanother vehicle Vre may be appropriately displayed in the combinedimage. Further, the feature 5w and the feature 7w are satisfied for theanother vehicle Vl2, and hence the another vehicle Vl2 may beappropriately displayed in the combined image.

Thus, in the case of Section 3-2, regardless of the angle (θn or θw) towhich the azimuth angle θl is changed, the target objects of the ownlane L and the left lane Ll can be appropriately displayed in thecombined image.

3-3. When Target Object of Left Lane Ll is Another Vehicle Vl3

The another vehicle Vl3 is positioned in the zone Z3. Accordingly, theCPU changes the azimuth angle θl to the angle θn (“narrow”) withreference to the matrix M1. As illustrated in FIG. 9A, in this case, thefeature 4n and the feature 6n are satisfied for the another vehicle Vre,and hence the another vehicle Vre may be appropriately displayed in thecombined image. Meanwhile, the feature 8n and the feature 10n aresatisfied for the another vehicle Vl3, and hence the another vehicle Vl3cannot be appropriately displayed in the combined image due to thedoubling phenomenon.

In contrast, when the azimuth angle θl is changed to the angle θw (seeFIG. 9B), the feature 4w and the feature 6w are satisfied for theanother vehicle Vre, and hence the another vehicle Vre may beappropriately displayed in the combined image. Meanwhile, the feature 8wand the feature 10w are satisfied for the another vehicle Vl3, but theanother vehicle Vl3 is positioned on the diagonally rearward left sideof the another vehicle Vre. Thus, depending on the positionalrelationship of both the vehicles, there is a possibility that thehiding phenomenon occurs.

In general, the doubling phenomenon is a problem that degrades theappearance of the combined image, while the hiding phenomenon is aproblem that has a possibility of causing reduction of the travelingsafety of the vehicle. Accordingly, in Section 3-3, priority is givenmore to solving the hiding phenomenon than to solving the doublingphenomenon. With this configuration, in the case of Section 3-3, theazimuth angle θl is changed to the angle θn so that the target objectsof the own lane L and the left lane Ll can be appropriately displayed inthe combined image.

3-4. When Target Object is Absent on Left Lane Ll

In this case, the CPU maintains the azimuth angle θl to the value in thecycle immediately before the current cycle (“no change”) with referenceto the matrix M1. When the azimuth angle θl in the cycle immediatelybefore the current cycle is the angle θn, the CPU maintains θl=θn in thecurrent cycle. In this case, as illustrated in FIG. 9A, the feature 4nand the feature 6n are satisfied for the another vehicle Vre, and hencethe another vehicle Vre may be appropriately displayed in the combinedimage.

Meanwhile, when the azimuth angle θl in the cycle immediately before thecurrent cycle is the angle θw, the CPU maintains θl=θw in the currentcycle. In this case, as illustrated in FIG. 9B, the feature 4w and thefeature 6w are satisfied for the another vehicle Vre, and hence theanother vehicle Vre may be appropriately displayed in the combinedimage.

Thus, in the case of Section 3-4, regardless of the angle (θn or θw) towhich the azimuth angle θl is changed, the target object of the own laneL can be appropriately displayed in the combined image.

Among the sixteen components of the matrix M1, the component of “nochange” may be set to any of “narrow” or “wide” in advance. The sameholds true for the matrix M2. With this configuration, the frequency ofswitching the combined image is increased, but the control content canbe simplified.

4. When Target Object of Own Lane L (Another Vehicle Vre) is Positionedin Zone Z1 (FIG. 10A and FIG. 10B)

4-1. When Target Object of Left Lane Ll is Another Vehicle Vl1

The another vehicle Vl1 is positioned in the zone Z1. Accordingly, theCPU changes the azimuth angle θl to the angle θn (“narrow”) withreference to the matrix M1. As illustrated in FIG. 10A, in this case,the feature in and the feature 3n are satisfied for the another vehicleVre, and hence the another vehicle Vre cannot be appropriately displayedin the combined image due to the cut-off phenomenon. Meanwhile, thefeature 2n and the feature 3n are satisfied for the another vehicle Vl1,and hence the another vehicle Vl1 may be appropriately displayed in thecombined image.

In contrast, when the azimuth angle θl is changed to the angle θw (seeFIG. 10B), the feature 1w and the feature 3w are satisfied for theanother vehicle Vre, and hence, unless the another vehicle Vre comesexcessively close to the own vehicle V, the another vehicle Vre may beappropriately displayed in the combined image. Meanwhile, the feature 2wand the feature 3w are satisfied for the another vehicle Vl1, and hencethe another vehicle Vl1 cannot be appropriately displayed in thecombined image due to the cut-off phenomenon. In some cases, thedisappearing phenomenon occurs depending on the size of the blind spotregion Rb. Further, when the target object of the left lane Ll is atwo-wheeled motor vehicle, the possibility that the disappearingphenomenon occurs is increased.

With reference to the feature in, when the cut-off phenomenon occurs inthe another vehicle Vre (that is, the target object of the own lane L),there is a high possibility that a left front corner portion (that is, apart in which a left turn signal lamp is installed) of the anothervehicle Vre is positioned in the blind spot region Rb. Accordingly, theleft turn signal lamp of the another vehicle Vre is not displayed in thecombined image, and the driver of the own vehicle V cannot determinefrom the combined image whether or not the another vehicle Vre intendsto change the lane. Thus, when the another vehicle Vre changes the laneto the left lane Ll at the timing at which the own vehicle V changes thelane to the left lane Ll, in some cases, the another vehicle Vre mayexcessively come close to or come into contact with the own vehicle Vdepending on the paths of both the vehicles. Thus, there is apossibility that the traveling safety of the own vehicle V is reduced.

Meanwhile, when the cut-off phenomenon or the disappearing phenomenonoccurs in the another vehicle Vl1, there is a possibility that thedriver cannot appropriately recognize the presence of the anothervehicle Vl1 from the combined image. When the own vehicle V changes thelane to the left lane Ll while the driver does not notice the presenceof the another vehicle Vl1, in some cases, the own vehicle V may comeinto contact with the another vehicle Vl1 traveling straight on the leftlane Ll. Thus, there is a possibility that the traveling safety of theown vehicle V is reduced.

In this case, with reference to the feature in, the blind spot region Rbis not so large, and hence the possibility that the disappearingphenomenon occurs in the another vehicle Vre is extremely low. That is,the cut-off phenomenon occurs in the another vehicle Vre, but the drivercan recognize the presence of the another vehicle Vre itself from thecombined image. In contrast, when the disappearing phenomenon occurs inthe another vehicle Vl1, the driver cannot recognize the presence of theanother vehicle Vl1 from the combined image. Accordingly, as compared tothe cut-off phenomenon of the target object of the own lane L (anothervehicle Vre), the cut-off or disappearing phenomenon of the targetobject of the left lane Ll (another vehicle Vl1) has a higherpossibility of affecting the traveling of the own vehicle V. Thus, inSection 4-1, priority is given more to solving the cut-off ordisappearing phenomenon of the target object of the left lane Ll than tosolving the cut-off phenomenon of the target object of the own lane L.With this configuration, in the case of Section 4-1, when the azimuthangle θl is changed to the angle θn, the target objects of the own laneL and the left lane Ll can be appropriately displayed in the combinedimage.

4-2. When Target Object of Left Lane Ll is Another Vehicle Vl2

The another vehicle Vl2 is positioned in the zone Z2. Accordingly, theCPU changes the azimuth angle θl to the angle θw (“wide”) with referenceto the matrix M1. As illustrated in FIG. 10B, in this case, the feature1w and the feature 3w are satisfied for the another vehicle Vre, andhence, unless the another vehicle Vre comes excessively close to the ownvehicle V, the another vehicle Vre may be appropriately displayed in thecombined image. Meanwhile, the feature 5w and the feature 7w aresatisfied for the another vehicle Vl2, but the another vehicle Vl2 ispositioned on the diagonally rearward left side of the another vehicleVre. Thus, there is a possibility that the hiding phenomenon occursdepending on the positional relationship of both the vehicles.

In contrast, when the azimuth angle θl is changed to the angle θn (seeFIG. 10A), the feature in and the feature 3n are satisfied for theanother vehicle Vre, and hence the another vehicle Vre cannot beappropriately displayed in the combined image due to the cut-offphenomenon. Meanwhile, the feature 5n and the feature 7n are satisfiedfor the another vehicle Vl2, and hence the another vehicle Vl2 may beappropriately displayed in the combined image.

Similarly to the case of Section 4-1 described above, when the cut-offphenomenon occurs in the another vehicle Vre, the driver cannotdetermine from the combined image whether or not the another vehicle Vreintends to change the lane. Accordingly, there is a possibility that thetraveling safety of the own vehicle V is reduced.

In general, the cut-off phenomenon has a higher possibility of affectingthe traveling of the own vehicle V as compared to the hiding phenomenon.Accordingly, in Section 4-2, priority is given more to solving thecut-off phenomenon than to solving the hiding phenomenon. With thisconfiguration, in the case of Section 4-2, when the azimuth angle θl ischanged to the angle θw, the target objects of the own lane L and theleft lane Ll can be appropriately displayed in the combined image.

4-3. When Target Object of Left Lane Ll is Another Vehicle Vl3

The another vehicle Vl3 is positioned in the zone Z3. Accordingly, theCPU changes the azimuth angle θl to the angle θw (“wide”) with referenceto the matrix M1. As illustrated in FIG. 10B, in this case, the feature1w and the feature 3w are satisfied for the another vehicle Vre, andhence, unless the another vehicle Vre comes excessively close to the ownvehicle V, the another vehicle Vre may be appropriately displayed in thecombined image. Meanwhile, the feature 8w and the feature 10w aresatisfied for the another vehicle Vl3, but the another vehicle Vl3 ispositioned on the diagonally rearward left side of the another vehicleVre. Thus, there is a possibility that the hiding phenomenon occursdepending on the positional relationship of both the vehicles.

In contrast, when the azimuth angle θl is changed to the angle θn (seeFIG. 10A), the feature in and the feature 3n are satisfied for theanother vehicle Vre, and hence the another vehicle Vre cannot beappropriately displayed in the combined image due to the cut-offphenomenon. Further, the feature 8n and the feature 10n are satisfiedfor the another vehicle Vl3, and hence the another vehicle Vl3 cannot beappropriately displayed in the combined image due to the doublingphenomenon.

Similarly to the case of Section 4-1 described above, when the cut-offphenomenon occurs in the another vehicle Vre, the driver cannotdetermine from the combined image whether or not the another vehicle Vreintends to change the lane. Accordingly, there is a possibility that thetraveling safety of the own vehicle V is reduced.

As described above, the cut-off phenomenon has a higher possibility ofaffecting the traveling of the own vehicle V as compared to the hidingphenomenon. Accordingly, in Section 4-3, priority is given more tosolving the cut-off phenomenon than to solving the hiding phenomenon.With this configuration, in the case of Section 4-3, when the azimuthangle θl is changed to the angle θw, the target objects of the own laneL and the left lane Ll can be appropriately displayed in the combinedimage.

4-4. When Target Object is Absent on Left Lane Ll

In this case, the CPU changes the azimuth angle θl to the angle θw(“wide”) with reference to the matrix M1. As illustrated in FIG. 10B, inthis case, the feature 1w and the feature 3w are satisfied for theanother vehicle Vre, and hence, unless the another vehicle Vre comesexcessively close to the own vehicle V, the another vehicle Vre may beappropriately displayed in the combined image.

In contrast, when the azimuth angle θl is changed to the angle θn (seeFIG. 10A), the feature in and the feature 3n are satisfied for theanother vehicle Vre, and hence the another vehicle Vre cannot beappropriately displayed in the combined image due to the cut-offphenomenon.

Thus, in the case of Section 4-4, when the azimuth angle θl is changedto the angle θw, the target object of the own lane L can beappropriately displayed in the combined image.

As described above, in the image control system 1, when the combinedimage is to be generated, the magnitude of the azimuth angle θl (or theazimuth angle θr) is changed based on the combination of “presence orabsence of a target object of the own lane L and a level of closeness tothis target object” and “presence or absence of a target object of theleft lane Ll (or a target object of the right lane Lr) and a level ofcloseness to this target object.” In addition, when the magnitude of theazimuth angle θl (or the azimuth angle θr) is changed, the magnitude ofthe azimuth angle θli (or the azimuth angle θri) is changed inaccordance with this change. When the magnitude of the azimuth angle θl(or the azimuth angle θr) and the magnitude of the azimuth angle θli (orthe azimuth angle θri) are appropriately changed independently of eachother based on this combination, an object having a relatively highpossibility of affecting the traveling of the vehicle (that is, a targetobject of the own lane L and a target object of the left lane Ll (or atarget object of the right lane Lr)) can be appropriately displayed inthe combined image.

Further, the change of the azimuth angle θl (or the azimuth angle θr)and the change of the azimuth angle θli (or the azimuth angle θri) areperformed based on only information on the target object of the own laneL and the target object of the left lane Ll (or the target object of theright lane Lr), and information on other objects is not considered.Thus, an increase of a processing load applied when the combined imageis generated can be suppressed.

In particular, in the at least one embodiment, the azimuth angle θl (orthe azimuth angle θr) can be changed between two types of angles, thatis, the angle θn (or −θn) and the angle θw (or −θw). Accordingly, with arelatively simple configuration, the target object of each of the lanesL, Ll, and Lr can be appropriately displayed in the combined image.

In the above, the image control system according to the at least oneembodiment have been described, but the present disclosure is notlimited to the above-mentioned at least one embodiment, and variouschanges are possible within the range not departing from the object ofthe present disclosure.

For example, as shown in FIG. 14A, the azimuth angle θl may bedetermined with reference to a matrix M3 serving as a modificationexample, in place of the matrix M1. Similarly, as shown in FIG. 14B, theazimuth angle θr may be determined with reference to a matrix M4 servingas another modification example, in place of the matrix M2. The matrixM3 (or M4) is different from the matrix M1 (or M2) in the followingpoints.

-   -   Among components of the matrix M3 (or M4), a component        corresponding to the component of “no change” in the matrix M1        (or M2) is set to “narrow.”    -   Among the components of the matrix M3 (or M4), a component        corresponding to a case in which the target object is absent on        the own lane L and the target object of the left lane Ll (or the        right lane Lr) is positioned in the zone Z3 (hereinafter        referred to as “component C1”) is set to “narrow.”    -   Among the components of the matrix M3 (or M4), a component        corresponding to a case in which both of the target object of        the own lane L and the target object of the left lane Ll (or the        right lane Lr) are positioned in the zone Z3 (hereinafter        referred to as “component C2”) is set to “narrow.”

With this configuration, among the sixteen components of the matrix M3(or M4), thirteen components are set to “narrow,” and hence thefrequency of changing the azimuth angle θl (or θr) (that is, thefrequency of moving the position of the seam of the combined image) isgreatly reduced. In this manner, the frequency of switching the combinedimage is reduced, and hence a possibility that the driver feels bothereddue to the frequent switching can be reduced. In addition, the frequencyat which the driver notices the switching and turns his or her eyes tothe display 12 a can be reduced. Meanwhile, when the component C1 andthe component C2 are set to “narrow,” the doubling phenomenon occurs inthe target object of the left lane Ll (see FIG. 7A and FIG. 8A).However, as described above, the doubling phenomenon merely degrades theappearance of the combined image, and hence the doubling phenomenon doesnot affect the traveling of the own vehicle V so much. That is, in thisconfiguration, priority is given more to “reducing the frequency ofswitching the combined image” than to “occurrence of the doublingphenomenon” (the occurrence of the doubling phenomenon is allowed).

With this configuration, even when the ECU 10 can calculate only thedistance to a relatively close object (typically, an object positionedin the zone Z1) (in other words, even when a computation limit distanceof the ECU 10 only includes the zone Z1), the azimuth angle θl (or θr)can be determined in accordance with the matrix M3 (or M4). That is, theECU 10 is configured to change the azimuth angle θl (or θr) to the angleθw (−θw) (“wide”) only in a case in which the target object of the ownlane L is positioned within the computation limit distance and thetarget object of the left lane Ll (or the right lane Lr) is notpositioned within the computation limit distance (including a case inwhich this target object is absent), and to change the azimuth angle θl(or θr) to the angle θn (−θn) (“narrow”) in other cases. In this manner,the present disclosure is applicable also to an image control system inwhich the computation limit distance of the ECU 10 does not reach thezone Z3.

Further, the ECU 10 may be configured to continuously change the azimuthangle θl (or θr) and the azimuth angle θli (or θri) when changing thoseazimuth angles. With this configuration, the movement of the seam of thecombined image becomes smoother, and hence the switching of the combinedimage can be performed smoothly (naturally). Thus, a possibility thatthe driver feels bothered by how the combined image is switched can bereduced.

Moreover, in a case in which the target object travels in the vicinityof a boundary between the zones Z1 and Z2 or in the vicinity of aboundary between the zones Z2 and Z3, when the position of the targetobject frequently changes between the zones Z1 and Z2 or between thezones Z2 and Z3, in some cases, the azimuth angle θl (or θr) may befrequently changed between the angles θn and θw (or between the angles−θn and −θw). In this case, the switching of the combined image isfrequently performed (chattering occurs), and there is a possibilitythat the driver feels bothered by how the combined image is switched. Inview of the above, the value of the distance D1 at the time when thetarget object enters the zone Z1 from the zone Z2 may be set to D1in(for example, 15 m), and the value of the distance D1 at the time whenthe target object enters the zone Z2 from the zone Z1 may be set to D1out (for example, 17 m) which is larger than D1 in. Similarly, the valueof the distance D2 at the time when the target object enters the zone Z2from the zone Z3 may be set to D2 in (for example, 30 m), and the valueof the distance D2 at the time when the target object enters the zone Z3from the zone Z2 may be set to D2out (for example, 34 m) which is largerthan D2 in. That is, hysteresis may be introduced to the distances D1and D2. In this manner, the occurrence of the chattering can besuppressed.

As another example, when the combined image is switched along with thechange of the azimuth angle θl (or θr), the ECU 10 may be configured toprevent the azimuth angle θ1 (or θr) from being changed (that is,prevent the combined image from being switched) until a predeterminedperiod (>T) elapses even when a condition for changing the azimuth angleθl (or θr) is satisfied during this period. Even with thisconfiguration, the occurrence of the chattering can be suppressed.

Moreover, the image control system 1 may include a distance measurementsensor, such as a radar, an ultrasonic sensor, or a laser radar. In thismanner, a computation accuracy of the distance to the object isimproved.

Moreover, the ECU 10 may be configured to display, in a part of thedisplay 12 a, a bird's-eye view (view of FIG. 5A or FIG. 5B) forillustrating the current effective ranges Rrea, Rla, and Rra. Thisdisplay may be constantly performed during a period in which theignition switch is turned on, or may be performed only when the combinedimage is switched.

Moreover, the ECU 10 may be configured to prevent the combined imagefrom being displayed on the display 12 a when the own vehicle V is inthe stop state or travels at low speed, and to display the combinedimage on the display 12 a only when the own vehicle V travels atrelatively middle-level vehicle speed or at high speed. The reason is asfollows. The case in which the own vehicle V is in the stop state ortravels at low speed is typically a case in which the own vehicle V iswaiting for a traffic light to change or the own vehicle V is caught ina traffic jam. In such a case, a possibility that the target object ofeach of the lanes L, Ll, and Lr affects the traveling of the own vehicleV is relatively low (that is, the necessity to display the combinedimage is not so high).

In this case, the ECU 10 may determine the azimuth angle θl (or θr) withreference to a matrix (not shown) in which a leftmost vertical column(that is, a column indicating components in a case in which the targetobject of the own lane L is positioned in the zone Z1) of the matrix M1(or M2) is deleted.

With this configuration, the configuration of the image control systemcan be simplified, and hence the cost can be reduced.

Moreover, the present disclosure can also be applied to a vehicletraveling by autonomous driving (autonomous driving control).

What is claimed is:
 1. An image control system, comprising: a rear imagepickup device configured to pick up images of an object and a dividingline which are present in a rear image pickup range expanding on a rearside of a vehicle; a left rear image pickup device configured to pick upimages of an object and a dividing line which are present in a left rearimage pickup range expanding on a left rear side of the vehicle andpartially overlapping the rear image pickup range; a right rear imagepickup device configured to pick up images of an object and a dividingline which are present in a right rear image pickup range expanding on aright rear side of the vehicle and partially overlapping the rear imagepickup range; a display device including a display screen; and an imagecontrol device configured to: acquire rear image data obtained bypicking up the images by the rear image pickup device, left rear imagedata obtained by picking up the images by the left rear image pickupdevice, and right rear image data obtained by picking up the images bythe right rear image pickup device; generate a combined image having apanorama format based on the rear image data, the left rear image data,and the right rear image data; and display the combined image on thedisplay screen of the display device, wherein, when, in the rear imagepickup range, the left rear image pickup range, and the right rear imagepickup range, image pickup ranges to be used for generating the combinedimage are defined as a rear effective range, a left rear effectiverange, and a right rear effective range, respectively, when, in a casein which a direction of an optical axis of the rear image pickup deviceis set as a rear reference direction, among boundary lines defining ahorizontal angle of view of the rear effective range, an angle from therear reference direction of a rear left-side boundary line being theboundary line on a left side of the vehicle is defined as a rearleft-side azimuth angle and an angle from the rear reference directionof a rear right-side boundary line being the boundary line on a rightside of the vehicle is defined as a rear right-side azimuth angle,respectively, when, in a case in which a direction of an optical axis ofthe left rear image pickup device is set as a left rear referencedirection, among boundary lines defining a horizontal angle of view ofthe left rear effective range, an angle from the left rear referencedirection of a left rear inner-side boundary line being the boundaryline on an inner side in a vehicle width direction of the vehicle isdefined as a left rear inner-side azimuth angle, and when, in a case inwhich a direction of an optical axis of the right rear image pickupdevice is set as a right rear reference direction, among boundary linesdefining a horizontal angle of view of the right rear effective range,an angle from the right rear reference direction of a right rearinner-side boundary line being the boundary line on an inner side in thevehicle width direction of the vehicle is defined as a right rearinner-side azimuth angle, the image control device is configured to:allow the rear effective range to be changed by changing magnitudes ofthe rear left-side azimuth angle and the rear right-side azimuth angleindependently of each other within a range of a horizontal angle of viewof the rear image pickup device; allow the left rear effective range tobe changed by changing a magnitude of the left rear inner-side azimuthangle within a range of a horizontal angle of view of the left rearimage pickup device; allow the right rear effective range to be changedby changing a magnitude of the right rear inner-side azimuth anglewithin a range of a horizontal angle of view of the right rear imagepickup device; and calculate, based on the dividing line detected fromeach of the rear image data, the left rear image data, and the rightrear image data, a position and a shape of each of an own lane on whichthe vehicle is positioned, a left lane adjacent to the own lane on theleft side thereof, and a right lane adjacent to the own lane on theright side thereof, and wherein, when the image control device is togenerate the combined image, the image control device is configured to:identify a rear target object, a left rear target object, and a rightrear target object which are objects closest to the vehicle in the ownlane, the left lane, and the right lane, respectively, among the objectspositioned on the rear side with respect to a rear end portion of thevehicle, which are detected from the rear image data, the left rearimage data, and the right rear image data; change the magnitude of therear left-side azimuth angle based on a combination of presence orabsence of the rear target object and a level of closeness to the reartarget object and presence or absence of the left rear target object anda level of closeness to the left rear target object; change themagnitude of the rear right-side azimuth angle based on a combination ofthe presence or absence of the rear target object and the level ofcloseness to the rear target object and presence or absence of the rightrear target object and a level of closeness to the right rear targetobject; change, when the magnitude of the rear left-side azimuth angleor the rear right-side azimuth angle is changed to a relatively smallvalue, the magnitude of the left rear inner-side azimuth angle or theright rear inner-side azimuth angle to a relatively large value,respectively, so as to generate the combined image; and change, when themagnitude of the rear left-side azimuth angle or the rear right-sideazimuth angle is changed to a relatively large value, the magnitude ofthe left rear inner-side azimuth angle or the right rear inner-sideazimuth angle to a relatively small value, respectively, so as togenerate the combined image.
 2. The image control system according toclaim 1, wherein the image control device is configured to: allow therear left-side azimuth angle to be changed between a first rearleft-side angle and a second rear left-side angle having a magnitudelarger than a magnitude of the first rear left-side angle; allow therear right-side azimuth angle to be changed between a first rearright-side angle and a second rear right-side angle having a magnitudelarger than a magnitude of the first rear right-side angle; allow theleft rear inner-side azimuth angle to be changed between a first leftrear inner-side angle and a second left rear inner-side angle having amagnitude larger than a magnitude of the first left rear inner-sideangle; and allow the right rear inner-side azimuth angle to be changedbetween a first right rear inner-side angle and a second right rearinner-side angle having a magnitude larger than a magnitude of the firstright rear inner-side angle.
 3. The image control system according toclaim 1, wherein the image control device is configured to: set animaginary projection surface orthogonal to a tread surface of thevehicle at a position separated away from the rear end portion of thevehicle rearward by a predetermined projection distance determined as afixed value; and generate, as the combined image, an image obtained byprojecting, onto the imaginary projection surface, the object and thedividing line present in each of the rear effective range, the left reareffective range, and the right rear effective range through use of aviewpoint as a reference, the viewpoint being imaginarily set on thevehicle or in vicinity thereof, and wherein, when the image controldevice is to generate the combined image, the image control device isconfigured to change each of the rear left-side azimuth angle, the leftrear inner-side azimuth angle, the rear right-side azimuth angle, andthe right rear inner-side azimuth angle so that a left-side intersectionand a right-side intersection are positioned on the imaginary projectionsurface in plan view of the vehicle, the left-side intersection being anintersection between the rear left-side boundary line and the left rearinner-side boundary line, the right-side intersection being anintersection between the rear right-side boundary line and the rightrear inner-side boundary line.
 4. The image control system according toclaim 3, wherein, when a region between a first straight line passingthrough the rear end portion of the vehicle and extending in the vehiclewidth direction of the vehicle and a second straight line separated awayfrom the first straight line rearward by a predetermined first distanceand in parallel to the first straight line is defined as a first zone, aregion between the second straight line and a third straight lineseparated away from the first straight line rearward by a predeterminedsecond distance and in parallel to the first straight line is defined asa second zone, and a region on the rear side with respect to the thirdstraight line is defined as a third zone, the predetermined projectiondistance is longer than the predetermined first distance and is shorterthan the predetermined second distance, and wherein, when anintersection between the imaginary projection surface and a leftimaginary line extending rearward from a left rear corner portion of thevehicle so as to be parallel to a longitudinal axis of the vehicle isdefined as a first intersection, and an intersection between theimaginary projection surface and a right imaginary line extendingrearward from a right rear corner portion of the vehicle so as to beparallel to the longitudinal axis of the vehicle is defined as a secondintersection, in a case in which the magnitude of the rear left-sideazimuth angle or the rear right-side azimuth angle is changed to therelatively small value, in plan view of the vehicle, the left-sideintersection or the right-side intersection is positioned on the firstintersection or in vicinity thereof, or on the second intersection or invicinity thereof, respectively, and in a case in which the magnitude ofthe rear left-side azimuth angle or the rear right-side azimuth angle ischanged to the relatively large value, the left-side intersection or theright-side intersection is positioned at a position separated away fromthe first intersection or the second intersection outward in the vehiclewidth direction of the vehicle, respectively, by a predetermineddistance equal to or larger than a general lane width.
 5. The imagecontrol system according to claim 4, wherein the predeterminedprojection distance is smaller than ½ of a sum of the predeterminedfirst distance and the predetermined second distance.
 6. The imagecontrol system according to claim 4, wherein the image control device isconfigured to: change the magnitude of the rear left-side azimuth angleto the relatively large value when the rear target object is positionedin the first zone and the left rear target object is positioned on therear side with respect to the first zone; and change the magnitude ofthe rear right-side azimuth angle to the relatively large value when therear target object is positioned in the first zone and the right reartarget object is positioned on the rear side with respect to the firstzone.
 7. The image control system according to claim 4, wherein theimage control device is configured to: change the magnitude of the rearleft-side azimuth angle to the relatively large value when the reartarget object is positioned in the first zone and the left rear targetobject is absent; and change the magnitude of the rear right-sideazimuth angle to the relatively large value when the rear target objectis positioned in the first zone and the right rear target object isabsent.
 8. The image control system according to claim 4, wherein theimage control device is configured to: change the magnitude of the rearleft-side azimuth angle to the relatively small value when both of therear target object and the left rear target object are positioned in thefirst zone; and change the magnitude of the rear right-side azimuthangle to the relatively small value when both of the rear target objectand the right rear target object are positioned in the first zone. 9.The image control system according to claim 4, wherein the image controldevice is configured to: change the magnitude of the rear left-sideazimuth angle to the relatively large value when both of the rear targetobject and the left rear target object are positioned in the third zone;change the magnitude of the rear right-side azimuth angle to therelatively large value when both of the rear target object and the rightrear target object are positioned in the third zone; change themagnitude of the rear left-side azimuth angle to the relatively largevalue when the rear target object is absent and the left rear targetobject is positioned in the third zone; and change the magnitude of therear right-side azimuth angle to the relatively large value when therear target object is absent and the right rear target object ispositioned in the third zone.
 10. The image control system according toclaim 1, wherein, when the image control device is to generate thecombined image, the image control device is configured to: change therear left-side azimuth angle and the left rear inner-side azimuth anglecontinuously when changing the rear left-side azimuth angle and the leftrear inner-side azimuth angle; and change the rear right-side azimuthangle and the right rear inner-side azimuth angle continuously whenchanging the rear right-side azimuth angle and the right rear inner-sideazimuth angle.
 11. The image control system according to claim 2,wherein the image control device is configured to: set an imaginaryprojection surface orthogonal to a tread surface of the vehicle at aposition separated away from the rear end portion of the vehiclerearward by a predetermined projection distance determined as a fixedvalue; and generate, as the combined image, an image obtained byprojecting, onto the imaginary projection surface, the object and thedividing line present in each of the rear effective range, the left reareffective range, and the right rear effective range through use of aviewpoint as a reference, the viewpoint being imaginarily set on thevehicle or in vicinity thereof, and wherein, when the image controldevice is to generate the combined image, the image control device isconfigured to change each of the rear left-side azimuth angle, the leftrear inner-side azimuth angle, the rear right-side azimuth angle, andthe right rear inner-side azimuth angle so that a left-side intersectionand a right-side intersection are positioned on the imaginary projectionsurface in plan view of the vehicle, the left-side intersection being anintersection between the rear left-side boundary line and the left rearinner-side boundary line, the right-side intersection being anintersection between the rear right-side boundary line and the rightrear inner-side boundary line.